We've had some interesting discussions about Web Services around the office
lately. One of the major challenges are people that want to consume Web Services
that transfer large amounts of data rather than a single operation per record or
having to deal with filter constraints. The major problem with this is
"ballooning" of data on the server - meaning you build up a huge result set to
send back to the client which can take up a ton of resources. This can increase
exponentially as the number of requests increases. One solution is to stream the
data to the clients in "chunks."
Yasser Shohoud has a
really good example of
streaming songs in this manner; however, many of us are dealing with data
from a database rather than songs so I wanted to blog about how this can be
done. First I'd like to thank someone who wishes to remain anonymous (we'll call
him Homer) for helping me understand conceptually what was happening during the
most challenging part of this example: ReadXml().
This example is architected with three assemblies: Client, Web Service, and a
Common library. Let's start with the Common library where our main functionality
will be a class named CustomerStream. CustomerStream simply implements
IXmlSerializable. This requires us to implement GetSchema, ReadXml, and WriteXml
methods. CustomerStream will be designed so that we can pass it a SqlDataReader
in the constructor so we can use that to write the data in chunks. First, we'll
implement GetSchema which is very simple, and WriteXml. WriteXml will use our
SqlDataReader to write out our data chunks (in this case we'll use Customer
records from the Northwind example database).
namespace StreamService.Common {
public class CustomerStream : IXmlSerializable {
private const string ns = "http://tempuri.org";
private SqlDataReader _reader;
public CustomerStream() { }
public CustomerStream(SqlDataReader reader) {
_reader = reader;
}
public System.Xml.Schema.XmlSchema GetSchema() {
string schema = @"<xs:schema id=""Customer"" targetNamespace=""http://tempuri.org/Customer.xsd"" elementFormDefault=""qualified"" xmlns=""http://tempuri.org/Customer.xsd"" xmlns:mstns=""http://tempuri.org/Customer.xsd"" xmlns:xs=""http://www.w3.org/2001/XMLSchema"">
<xs:element name=""Customer"">
<xs:complexType>
<xs:sequence>
<xs:element name=""CustomerID"" type=""xs:string"" />
<xs:element name=""CompanyName"" type=""xs:string"" />
<xs:element name=""ContactName"" type=""xs:string"" />
<xs:element name=""Country"" type=""xs:string"" />
<xs:element name=""Phone"" type=""xs:string"" />
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:schema>";
XmlSchema xs = XmlSchema.Read(new StringReader(schema), null);
return xs;
}
public void WriteXml(System.Xml.XmlWriter writer) {
if (_reader == null) throw new InvalidOperationException("SqlDataReader expected");
using (_reader) {
while (_reader.Read()) {
writer.WriteStartElement("Customer", ns);
for (int fieldIndex = 0; fieldIndex < _reader.FieldCount; fieldIndex++) {
writer.WriteElementString(_reader.GetName(fieldIndex), ns, _reader.GetValue(fieldIndex).ToString());
}
writer.WriteEndElement();
}
}
}
}
}
The next class we'll want to implement is some sort of custom reader for
iterating the results. Because this is a dependency of the ReadXml() method
which we haven't implemented at this point, we'll do that first. This is a very
simplistic design that consists mostly of throwing NotImplementedException (haha),
but gets the job done for this example and you could certainly extend this
concept. Basically the custructor takes a file path where the results are stored
in an XML document and operates similiar to a datareader using GetCustomer().
namespace StreamService.Common {
public class CustomerDataReader : DbDataReader, IDataReader, IDisposable, IDataRecord {
private Dictionary<int, string> _ordinals;
private XmlNodeList _nodes;
private XmlDocument _document;
private Customer _current;
private int _count;
public CustomerDataReader(string path) {
_document = new XmlDocument();
_document.Load(path);
_nodes = _document.SelectNodes("//Customers/*");
_ordinals = new Dictionary<int, string>();
_ordinals.Add(0, "CustomerID");
_ordinals.Add(1, "CompanyName");
_ordinals.Add(2, "ContactName");
_ordinals.Add(3, "Country");
_ordinals.Add(4, "Phone");
}
public override int GetOrdinal(string name) {
foreach (KeyValuePair<int, string> entry in _ordinals) {
if (entry.Value == name) return entry.Key;
}
throw new ArgumentException("Name does not exist.");
}
public override bool Read() {
if (_count < _nodes.Count) {
XmlNode currentNode = _nodes[_count++];
_current = new Customer();
foreach (XmlNode child in currentNode.ChildNodes) {
typeof(Customer).GetProperty(child.Name).SetValue(_current, child.InnerText, null);
}
return true;
}
return false;
}
public Customer GetCustomer() {
return _current;
}
public override void Close() {
// ...
}
// NOTE: rest of the methods throw not implemented exceptions
}
}
Now we are set to implement the ReadXml() method that is missing from our
CustomerStream class as well as a method that allows the calling client to
implement our custom datareader class...
//CustomerStream class methods...
public CustomerDataReader ExecuteCustomerDataReader() {
CustomerDataReader reader = new CustomerDataReader(Environment.CurrentDirectory + "\\Customers.xml");
return reader;
}
public void ReadXml(System.Xml.XmlReader reader) {
using (XmlTextWriter xwriter = new XmlTextWriter(Environment.CurrentDirectory + "\\Customers.xml", Encoding.UTF8)) {
xwriter.WriteRaw(@"<?xml version=""1.0"" encoding=""utf-8""?>");
xwriter.WriteStartElement("Customers");
while (reader.Read()) {
if (reader.LocalName == "Customer") {
xwriter.WriteStartElement("Customer");
reader.ReadStartElement();
while (reader.NodeType != XmlNodeType.EndElement) {
xwriter.WriteElementString(reader.LocalName, reader.ReadElementString());
}
xwriter.WriteEndElement();
}
}
xwriter.WriteEndElement();
}
}
Since our custom datareader class exposes a class type named Customer, here
is the code for that...
namespace StreamService.Common {
public class Customer {
private string _phone;
private string _country;
private string _contactName;
private string _companyName;
private string _customerID;
public Customer() { }
public Customer(string customerID, string companyName, string contactName, string country, string phone) {
_customerID = customerID;
_companyName = companyName;
_contactName = contactName;
_country = country;
_phone = phone;
}
public string CustomerID {
get { return _customerID; }
set { _customerID = value; }
}
public string CompanyName {
get { return _companyName; }
set { _companyName = value; }
}
public string ContactName {
get { return _contactName; }
set { _contactName = value; }
}
public string Country {
get { return _country; }
set { _country = value; }
}
public string Phone {
get { return _phone; }
set { _phone = value; }
}
}
}
The next step is to write our Web Service. This is very straight forward -
all we have to do is turn off buffering and create a SqlDataReader to query the
Customers table in the Northwind database. Note that the responsibility for
closing the SqlDataReader and the SqlConnection lies within the CustomerStream
class. This is so that we do not get disconnected after the GetCustomers
WebMethod finishes executing and we are within the context of the server
streaming down the data chunks (customer records) to the client...
namespace StreamService.Server {
[WebService(Namespace = "http://tempuri.org/")]
[WebServiceBinding(ConformsTo = WsiProfiles.BasicProfile1_1)]
[System.ComponentModel.DesignerCategory("Code")]
[ToolboxItem(false)]
public class TheWebService : System.Web.Services.WebService {
[WebMethod]
public CustomerStream GetCustomers() {
//turn off buffering
HttpContext.Current.Response.Buffer = false;
//do *not* dispose of connection or datareader - our CustomerStream class does this
SqlConnection connection = new SqlConnection(WebConfigurationManager.ConnectionStrings["Northwind"].ConnectionString);
using (SqlCommand command = new SqlCommand("SELECT CustomerID, CompanyName, ContactName, Country, Phone FROM Customers", connection)) {
connection.Open();
SqlDataReader reader = command.ExecuteReader(CommandBehavior.CloseConnection);
CustomerStream stream = new CustomerStream(reader);
return stream;
}
}
}
}
Next, we need to create a client application to call the Web Service...
namespace StreamService.Client {
class Program {
static void Main(string[] args) {
TheWebService ws = new TheWebService();
CustomerStream stream = ws.GetCustomers();
XmlSerializer ser = new XmlSerializer(typeof(Customer));
using(CustomerDataReader reader = stream.ExecuteCustomerDataReader()){
while (reader.Read()) {
Customer customer = reader.GetCustomer();
ser.Serialize(Console.Out, customer);
}
}
}
}
}
Lastly, we'll need to make a small tweak to the client's Reference.cs code
(our proxy class generated by Visual Studio when we reference the Web Service)
so that the type casted and returned is a CustomerStream object..
/// <remarks/>
[System.Web.Services.Protocols.SoapDocumentMethodAttribute("http://tempuri.org/GetCustomers", RequestNamespace="http://tempuri.org/", ResponseNamespace="http://tempuri.org/", Use=System.Web.Services.Description.SoapBindingUse.Literal, ParameterStyle=System.Web.Services.Protocols.SoapParameterStyle.Wrapped)]
public CustomerStream GetCustomers() {
object[] results = this.Invoke("GetCustomers", new object[0]);
return ((CustomerStream)(results[0]));
}
I hope you found this interesting and informative. Merry Christmas, Happy
Holidays, etc!