Microsoft.Diagnostics.FastSerialization

Allows users of serialization and de-serialization mechanisms to specify the size of the StreamLabel. As traces get larger, there is a need to support larger file sizes, and thus to increase the addressable space within the files. StreamLabel instances are 8-bytes in-memory, but all serialization and de-serialization of them results in the upper 4-bytes being lost. This setting will allow Serializer and Deserializer to read and write 8-byte StreamLabel instances.

Allows users of serialization and de-serialization mechanism to specify the alignment required by the reader.

These settings apply to use of Serializer and Deserializer specifically.

A StreamLabel represents a position in a IStreamReader or IStreamWriter. In memory it is represented as a 64 bit signed value but to preserve compat with the FastSerializer.1 format it is a 32 bit unsigned value when serialized in a file. FastSerializer can parse files exceeding 32 bit sizes as long as the format doesn't persist a StreamLabel in the content. NetTrace is an example of this. During writing it is generated by the IStreamWriter.GetLabel method an consumed by the IStreamWriter.WriteLabel method. On reading you can use IStreamReader.Current and and IStreamReader.

Represents a stream label that is not a valid value

IStreamWriter is meant to be a very simple streaming protocol. You can write integral types, strings, and labels to the stream itself. IStreamWrite can be thought of a simplified System.IO.BinaryWriter, or maybe the writer part of a System.IO.Stream with a few helpers for primitive types. See also IStreamReader

Write a byte to a stream

Write a short to a stream

Write an int to a stream

Write a long to a stream

Write a StreamLabel (a pointer to another part of the stream) to a stream

Write a string to a stream (supports null values).

Get the stream label for the current position (points at whatever is written next

Write a SuffixLabel it must be the last thing written to the stream. The stream guarantees that this value can be efficiently read at any time (probably by seeking back from the end of the stream)). The idea is that when you generate a 'tableOfContents' you can only do this after processing the data (and probably writing it out), If you remember where you write this table of contents and then write a suffix label to it as the last thing in the stream using this API, you guarantee that the reader can efficiently seek to the end, read the value, and then goto that position. (See IStreamReader.GotoSuffixLabel for more)

IStreamReader is meant to be a very simple streaming protocol. You can read integral types, strings, and labels to the stream itself. You can also goto labels you have read from the stream. IStreamReader can be thought of a simplified System.IO.BinaryReder, or maybe the reader part of a System.IO.Stream with a few helpers for primitive types. See also IStreamWriter

Read a byte from the stream

Read a short from the stream

Read an int from the stream

Read a long from the stream

Read a string from the stream. Can represent null strings

Read a span of bytes from the stream.

Read a StreamLabel (pointer to some other part of the stream) from the stream

Goto a location in the stream

Returns the current position in the stream.

Sometimes information is only known after writing the entire stream. This information can be put on the end of the stream, but there needs to be a way of finding it relative to the end, rather than from the beginning. A IStreamReader, however, does not actually let you go 'backwards' easily because it does not guarantee the size what it writes out (it might compress). The solution is the concept of a 'suffixLabel' which is location in the stream where you can always efficiently get to. It is written with a special API (WriteSuffixLabel that must be the last thing written. It is expected that it simply write an uncompressed StreamLabel. It can then be used by using the GotoSTreamLabel() method below. This goes to this well know position in the stream. We expect this is implemented by seeking to the end of the stream, reading the uncompressed streamLabel, and then seeking to that position.

Support for higher level operations on IStreamWriter and IStreamReader

Writes a Guid to stream 'writer' as sequence of 8 bytes

Reads a Guid to stream 'reader' as sequence of 8 bytes and returns it

Returns a StreamLabel that is the sum of label + offset.

Returns the difference between two stream labels

Convenience method for skipping a a certain number of bytes in the stream.

Like a StreamLabel, a ForwardReference represents a pointer to a location in the stream. However unlike a StreamLabel, the exact value in the stream does not need to be known at the time the forward references is written. Instead the ID is written, and later that ID is associated with the target location (using DefineForwardReference).

Returned when no appropriate ForwardReference exists.

#SerializerIntroduction see also #StreamLayout The Serializer class is a general purpose object graph serializer helper. While it does not have any knowledge of the serialization format of individual object, it does impose conventions on how to serialize support information like the header (which holds versioning information), a trailer (which holds deferred pointer information), and how types are versioned. However these conventions are intended to be very generic and thus this class can be used for essentially any serialization need. Goals: * Allows full range of serialization, including subclassing and cyclic object graphs. * Can be serialized and deserialized efficiently sequentially (no seeks MANDATED on read or write). This allows the serializer to be used over pipes and other non-seekable devices). * Pay for play (thus very efficient in simple cases (no subclassing or cyclic graphs). * Ideally self-describing, and debuggable (output as XML if desired?) Versioning: * We want the ability for new formats to accept old versions if objects wish to support old formats * Also wish to allow new formats to be read by OLD version if the new format is just an 'extension' (data added to end of objects). This makes making new versions almost pain-free. Concepts: * No-seek requirement The serialized form should be such that it can be deserialized efficiently in a serial fashion (no seeks). This means all information needed to deserialize has to be 'just in time' (can't be some table at the end). Pragmatically this means that type information (needed to create instances), has to be output on first use, so it is available for the deserializer. * Laziness requirement While is should be possible to read the serialized for sequentially, we should also not force it. It should be possible to have a large file that represents a persisted structure that can be lazily brought into memory on demand. This means that all information needed to deserialize must also be 'randomly available' and not depend on reading from the beginning. Pragmatically this means that type information, and forward forwardReference information needs to have a table in a well known Location at the end so that it can be found without having to search the file sequentially. * Versioning requirement To allow OLD code to access NEW formats, it must be the case that the serialized form of every instance knows how to 'skip' past any new data (even if it does not know its exact size). To support this, objects have 'begin' and 'end' tags, which allows the deserializer to skip the next object. * Polymorphism requirement Because the user of a filed may not know the exact instance stored there, in general objects need to store the exact type of the instance. Thus they need to store a type identifier, this can be folded into the 'begin' tag. * Arbitrary object graph (circularity) requirement (Forward references) The serializer needs to be able to serialize arbitrary object graphs, including those with cycles in them. While you can do this without forward references, the system is more flexible if it has the concept of a forward reference. Thus whenever a object reference is required, a 'forward forwardReference' can be given instead. What gets serialized is simply an unique forward reference index (index into an array), and at some later time that index is given its true value. This can either happen with the target object is serialized (see Serializer.Tags.ForwardDefintion) or at the end of the serialization in a forward reference table (which allows forward references to be resolved without scanning then entire file. * Contract between objects IFastSerializable.ToStream: The heart of the serialization and deserialization process the IFastSerializable interface, which implements just two methods: ToStream (for serializing an object), and FromStream (for deserializing and object). This interfaces is the mechanism by which objects tell the serializer what data to store for an individual instance. However this core is not enough. An object that implements IFastSerializable must also implement a default constructor (constructor with no args), so that that deserializer can create the object (and then call FromStream to populated it). The ToStream method is only responsible for serializing the data in the object, and by itself is not sufficient to serialize an interconnected, polymorphic graph of objects. It needs help from the Serializer and Deserialize to do this. Serializer takes on the responsibility to deal with persisting type information (so that Deserialize can create the correct type before IFastSerializable.FromStream is called). It is also the serializer's responsibility to provide the mechanism for dealing with circular object graphs and forward references. * Layout of a serialized object: A serialized object has the following basic format * If the object is the definition of a previous forward references, then the definition must begin with a Serializer.Tags.ForwardDefintion tag followed by a forward forwardReference index which is being defined. * Serializer.Tags.BeginObject tag * A reference to the SerializationType for the object. This reference CANNOT be a forward forwardReference because its value is needed during the deserialization process before forward references are resolved. * All the data that that objects 'IFastSerializable.ToStream method wrote. This is the heart of the deserialized data, and the object itself has a lot of control over this format. * Serializer.Tags.EndObject tag. This marks the end of the object. It quickly finds bugs in ToStream FromStream mismatches, and also allows for V1 deserializers to skip past additional fields added since V1. * Serializing Object references: When an object forwardReference is serialized, any of the following may follow in the stream * Serializer.Tags.NullReference used to encode a null object forwardReference. * Serializer.Tags.BeginObject or Serializer.Tags.ForwardDefintion, which indicates that this the first time the target object has been referenced, and the target is being serialized on the spot. * Serializer.Tags.ObjectReference which indicates that the target object has already been serialized and what follows is the StreamLabel of where the definition is. * Serializer.Tags.ForwardReference followed by a new forward forwardReference index. This indicates that the object is not yet serialized, but the serializer has chosen not to immediately serialize the object. Ultimately this object will be defined, but has not happened yet. * Serializing Types: Types are simply objects of type SerializationType which contain enough information about the type for the Deserializer to do its work (it full name and version number). They are serialized just like all other types. The only thing special about it is that references to types after the BeginObject tag must not be forward references. #StreamLayout: The structure of the file as a whole is simply a list of objects. The first and last objects in the file are part of the serialization infrastructure. Layout Synopsis * Signature representing Serializer format * EntryObject (most of the rest of the file) * BeginObject tag * Type for This object (which is a object of type SerializationType) * BeginObject tag * Type for SerializationType POSITION1 * BeginObject tag * Type for SerializationType * ObjectReference tag // This is how our recursion ends. * StreamLabel for POSITION1 * Version Field for SerializationType * Minimum Version Field for SerializationType * FullName string for SerializationType * EndObject tag * Version field for EntryObject's type * Minimum Version field for EntryObject's type * FullName string for EntryObject's type * EndObject tag * Field1 * Field2 * V2_Field (this should be tagged so that it can be skipped by V1 deserializers. * EndObject tag * ForwardReferenceTable pseudo-object * Count of forward references * StreamLabel for forward ref 0 * StreamLabel for forward ref 1. * ... * SerializationTrailer pseudo-object * StreamLabel ForwardReferenceTable * StreamLabel to SerializationTrailer * End of stream

Create a serializer writes 'entryObject' to a file.

Create a serializer that writes to a . The serializer will close the stream when it closes.

Create a serializer that writes to a . The parameter determines whether the serializer will close the stream when it closes.

Create a serializer that writes 'entryObject' another IStreamWriter

Write a bool to a stream

Write a byte to a stream

Write a short to a stream

Write an int to a stream

Write a long to a stream

Write a Guid to a stream

Write a string to a stream

Write a float to a stream

Write a double to a stream

Write a StreamLabel (pointer to some other part of the stream whose location is current known) to the stream

Write a ForwardReference (pointer to some other part of the stream that whose location is not currently known) to the stream

If the object is potentially aliased (multiple references to it), you should write it with this method.

To tune working set (or disk seeks), or to make the dump of the format more readable, it is valuable to have control over which of several references to an object will actually cause it to be serialized (by default the first encountered does it). WriteDefered allows you to write just a forwardReference to an object with the expectation that somewhere later in the serialization process the object will be serialized. If no call to WriteObject() occurs, then the object is serialized automatically before the stream is closed (thus dangling references are impossible).

This is an optimized version of WriteObjectReference that can be used in some cases. If the object is not aliased (it has an 'owner' and only that owner has references to it (which implies its lifetime is strictly less than its owners), then the serialization system does not need to put the object in the 'interning' table. This saves a space (entries in the intern table as well as 'SyncEntry' overhead of creating hash codes for object) as well as time (to create that bookkeeping) for each object that is treated as private (which can add up if because it is common that many objects are private). The private instances are also marked in the serialized format so on reading there is a similar bookkeeping savings. The ultimate bits written by WritePrivateObject are the same as WriteObject. TODO Need a DEBUG mode where we detect if others besides the owner reference the object.

Create a ForwardReference. At some point before the end of the serialization, DefineForwardReference must be called on this value

Define the ForwardReference forwardReference to point at the current write location.

Write a byte preceded by a tag that indicates its a byte. These should be read with the corresponding TryReadTagged operation

Write a byte preceded by a tag that indicates its a short. These should be read with the corresponding TryReadTagged operation

Write a byte preceded by a tag that indicates its a int. These should be read with the corresponding TryReadTagged operation

Write a byte preceded by a tag that indicates its a long. These should be read with the corresponding TryReadTagged operation

Write a byte preceded by a tag that indicates its a string. These should be read with the corresponding TryReadTagged operation

Write a byte preceded by a tag that indicates its a object. These should be read with the corresponding TryReadTagged operation

Writes the header for a skipping an arbitrary blob. THus it writes a Blob tag and the size, and the caller must then write 'sizes' bytes of data in some way. This allows you to create regions of arbitrary size that can be skipped by old as well as new parsers.

Writes an end tag (which is different from all others). This is useful when you have a deferred region of tagged items.

Retrieve the underlying stream we are writing to. Generally the Write* methods are enough.

Completes the writing of the stream.

To help debug any serialization issues, you can write data to a side file called 'log.serialize.xml' which can track exactly what serialization operations occurred.

Dispose pattern

Deserializer is a helper class that holds all the information needed to deserialize an object graph as a whole (things like the table of objects already deserialized, and the list of types in the object graph. see #SerializerIntroduction for more

Create a Deserializer that reads its data from a given file

Create a Deserializer that reads its data from a given System.IO.Stream. The stream will be closed when the Deserializer is done with it.

Create a Deserializer that reads its data from a given System.IO.Stream. The parameter determines whether the deserializer will close the stream when it closes.

Create a Deserializer that reads its data from a given IStreamReader. The stream will be closed when the Deserializer is done with it.

Returns the full name of the type of the entry object without actually creating it. Will return null on failure.

GetEntryObject is the main deserialization entry point. The serialization stream always has an object that represents the stream as a whole, called the entry object and this returns it and places it in 'ret'

GetEntryObject is the main deserialization entry point. The serialization stream always has an object that represents the stream as a whole, called the entry object and this returns it and returns it

Read a bool from the stream

Read a byte from the stream

Read a short from the stream

Read an int from the stream

Read a long from the stream

Read a Guid from the stream

Read a float from the stream

Read a double from the stream

Read a string from the stream. Can represent null

d) from the stream

Read a IFastSerializable object from the stream and place it in ret

Read a IFastSerializable object from the stream and return it

Read a bool from the stream and return it

Read a byte from the stream and return it

Read a short from the stream and return it

Read an int from the stream and return it

Read a long from the stream and return it

Read a float from the stream and return it

Read a double from the stream and return it

Read in a string value and return it

Read in a StreamLabel (a pointer to some other part of the stream) and return it

Read in a ForwardReference (a pointer to some other part of the stream which was not known at the tie it was written) and return it Use ResolveForwardReference to convert the ForwardReference to a StreamLabel

Given a forward reference find the StreamLabel (location in the stream) that it points at). Normally this call preserves the current read location, but if you do don't care you can set preserveCurrent as an optimization to make it more efficient.

Meant to be called from FromStream. It returns the version number of the type being deserialized. It can be used so that new code can recognizes that it is reading an old file format and adjust what it reads.

Meant to be called from FromStream. It returns the version number of the MinimumReaderVersion of the type that was serialized.

The filename if read from a file or the stream name if read from a stream

If set this function is set, then it is called whenever a type name from the serialization data is encountered. It is your you then need to look that up. If it is not present it uses Type.GetType(string) which only checks the current assembly and mscorlib.

For every IFastSerializable object being deserialized, the Deserializer needs to create 'empty' objects that 'FromStream' is invoked on. The Deserializer gets these 'empty' objects by calling a 'factory' delegate for that type. Thus all types being deserialized must have a factory. RegisterFactory registers such a factory for particular 'type'.

For every IFastSerializable object being deserialized, the Deserializer needs to create 'empty' objects that 'FromStream' is invoked on. The Deserializer gets these 'empty' objects by calling a 'factory' delegate for that type. Thus all types being deserialized must have a factory. RegisterDefaultFactory registers a factory that is passed a type parameter and returns a new IFastSerialable object.

Try to read tagged value from the stream. If it is a tagged bool, return int in ret and return true, otherwise leave the cursor unchanged and return false

Try to read tagged value from the stream. If it is a tagged byte, return int in ret and return true, otherwise leave the cursor unchanged and return false

Try to read tagged value from the stream. If it is a tagged short, return int in ret and return true, otherwise leave the cursor unchanged and return false

Try to read tagged value from the stream. If it is a tagged int, return int in ret and return true, otherwise leave the cursor unchanged and return false

Try to read tagged value from the stream. If it is a tagged long, return int in ret and return true, otherwise leave the cursor unchanged and return false

Try to read tagged value from the stream. If it is a tagged string, return int in ret and return true, otherwise leave the cursor unchanged and return false

Try to read the header for a tagged blob of bytes. If Current points at a tagged blob it succeeds and returns the size of the blob (the caller must read or skip past it manually) If it is not a tagged blob it returns a size of 0 and resets the read pointer to what it was before this method was called.

Try to read tagged value from the stream. If it is a tagged FastSerializable, return int in ret and return true, otherwise leave the cursor unchanged and return false

Try to read tagged value from the stream. If it is a tagged FastSerializable, return it, otherwise leave the cursor unchanged and return null

Set the read position to the given StreamLabel

Set the read position to the given ForwardReference

Returns the current read position in the stream.

Fetch the underlying IStreamReader that the deserializer reads data from

Close the IStreamReader and free resources associated with the Deserializer

When we encounter a forward reference, we can either go to the forward reference table immediately and resolve it (deferForwardReferences == false), or simply remember that that position needs to be fixed up and continue with the deserialization. This later approach allows 'no seek' deserialization. This variable which scheme we do.

#DeferedRegionOverview. A DeferedRegion help make 'lazy' objects. You will have a DeferedRegion for each block of object you wish to independently decide whether to deserialize lazily (typically you have one per object however in the limit you can have one per field, it is up to you). When you call DeferedRegion.Write you give it a delegate that will write all the deferred fields. The Write operation will place a forward reference in the stream that skips all the fields written, then the fields themselves, then define the forward reference. This allows readers to skip the deferred fields. When you call DeferedRegion.Read you also give it a delegate that reads all the deferred fields. However when 'Read' instead of reading the fields it * remembers the deserializer, stream position, and reading delegate. * it uses the forward reference to skip the region. When DeferedRegion.FinishRead is called, it first checks if the region was already restored. If not it used the information to read in the deferred region and returns. Thus this FinishRead should be called before any deferred field is used.

see #DeferedRegionOverview. TODO more

See overview in DeferedRegion class comment. This call indicates that the 'fromStream' delegate can deserialize a region of the object, which was serialized with the DeferedRegion.Write method. The read skips the data for the region (thus no objects associated with the region are created in memory) but the deferred object remembers 'fromStream' and will call it when 'FinishRead()' is called.

FinishRead indicates that you need to deserialize the lazy region you defined with the 'Read' method. If the region has already been deserialized, nothing is done. Otherwise when you call this method the current position in the stream is put back to where it was when Read was called and the 'fromStream' delegate registered in 'Read' is called to perform the deserialization.

Returns true if the FinsihRead() has already been called.

Get the deserializer associated with this DeferredRegion

Get the stream position when Read was called

This helper is just here to ensure that FinishRead gets inlined

A type can opt into being serializable by implementing IFastSerializable and a default constructor (constructor that takes not arguments). Conceptually all clients of IFastSerializable also implement IFastSerializableVersion however the serializer will assume a default implementation of IFastSerializableVersion (that Returns version 1 and assumes all versions are allowed to deserialize it.

Given a Serializer, write yourself to the output stream. Conceptually this routine is NOT responsible for serializing its type information but only its field values. However it is conceptually responsible for the full transitive closure of its fields. * For primitive fields, the choice is easy, simply call Serializer.Write * For object fields there is a choice * If is is only references by the enclosing object (eg and therefore field's lifetime is identical to referencing object), then the Serialize.WritePrivateObject can be used. This skips placing the object in the interning table (that ensures it is written exactly once). * Otherwise call Serialize.WriteObject * For value type fields (or collections of structs), you serialize the component fields. * For collections, typically you serialize an integer inclusiveCountRet followed by each object.

Given a reader, and a 'this' instance, made by calling the default constructor, create a fully initialized instance of the object from the reader stream. The deserializer provides the extra state needed to do this for cyclic object graphs. Note that it is legal for the instance to cache the deserializer and thus be 'lazy' about when the actual deserialization happens (thus large persisted strucuture on the disk might stay on the disk). Typically the FromStream implementation is an exact mirror of the ToStream implementation, where there is a Read() for every Write().

Objects implement IFastSerializableVersion to indicate what the current version is for writing and which readers can read the current version. If this interface is not implemented a default is provided (assuming version 1 for writing and MinimumVersion = 0). By default Serializer.WriteObject will place marks when the object ends and always skip to the end even if the FromStream did not read all the object data. This allows considerable versioning flexibility. Simply by placing the new data at the end of the existing serialization, new versions of the type can be read by OLD deserializers (new fields will have the value determined by the default constructor (typically 0 or null). This makes is relatively easy to keep MinimumVersion = 0 (the ideal case).

This is the version number for the serialization CODE (that is the app decoding the format) It should be incremented whenever a change is made to IFastSerializable.ToStream and the format is publicly disseminated. It must not vary from instance to instance. This is pretty straightforward. It defaults to 0

At some point typically you give up allowing new versions of the read to read old wire formats This is the Minimum version of the serialized data that this reader can deserialize. Trying to read wire formats strictly smaller (older) than this will fail. Setting this to the current version indicates that you don't care about ever reading data generated with an older version of the code. If you set this to something other than your current version, you are obligated to ensure that your FromStream() method can handle all formats >= than this number. You can achieve this if you simply use the 'WriteTagged' and 'ReadTagged' APIs in your 'ToStream' and 'FromStream' after your V1 AND you always add new fields to the end of your class. This is the best practice. Thus void IFastSerializable.ToStream(Serializer serializer) { serializer.Write(Ver_1_Field1); serializer.Write(Ver_1_Field2); // ... serializer.WriteTagged(Ver_2_Field1); serializer.WriteTagged(Ver_2_Field2); // ... serializer.WriteTagged(Ver_3_Field1); } void IFastSerializable.FromStream(Deserializer deserializer) { deserializer.Read(out Ver_1_Field1); deserializer.Read(out Ver_1_Field2); // ... deserializer.TryReadTagged(ref Ver_2_Field1); // If data no present (old format) then Ver_2_Field1 not set. deserializer.TryReadTagged(ref Ver_2_Field2); // ditto... // ... deserializer.TryReadTagged(ref Ver_3_Field1); } Tagging outputs a byte tag in addition to the field itself. If that is a problem you can also use the VersionBeingRead to find out what format is being read and write code that explicitly handles it. Note however that this only gets you Backward compatibility (new readers can read the old format, but old readers will still not be able to read the new format), which is why this is not the preferred method. void IFastSerializable.FromStream(Deserializer deserializer) { // We assume that MinVersionCanRead == 4 // Deserialize things that are common to all versions (4 and earlier) if (deserializer.VersionBeingRead >= 5) { deserializer.Read(AVersion5Field); if (deserializer.VersionBeingRead >= 5) deserializer.ReadTagged(AVersion6Field); } }

This is the minimum version of a READER that can read this format. If you don't support forward compatibility (old readers reading data generated by new readers) then this should be set to the current version. If you set this to something besides the current version you are obligated to ensure that your ToStream() method ONLY adds fields at the end, AND that all of those added fields use the WriteTagged() operations (which tags the data in a way that old readers can skip even if they don't know what it is) In addition your FromStream() method must read these with the ReadTagged() deserializer APIs. See the comment in front of MinimumVersionCanRead for an example of using the WriteTagged() and ReadTagged() methods.

Thrown when the deserializer detects an error.

Thown when a error occurs in serialization.

This is the version represents the version of both the reading code and the version for the format for this type in serialized form. See IFastSerializableVersion for more.

The version the the smallest (oldest) reader code that can read this file format. Readers strictly less than this are rejected. This allows support for forward compatbility. See IFastSerializableVersion for more.

Create a IStreamReader (reads binary data) from a given byte buffer

Create a IStreamReader (reads binary data) from a given subregion of a byte buffer

The total length of bytes that this reader can read.

Implementation of IStreamReader

Dispose pattern

Returns the SerializationConfiguration for this stream reader.

Returns the SerializationConfiguration for this stream writer.

A MemoryStreamReader is an implementation of the IStreamReader interface that works over a given byte[] array.

Create a IStreamReader (reads binary data) from a given byte buffer

Create a IStreamReader (reads binary data) from a given subregion of a byte buffer.

The total length of bytes that this reader can read.

Implementation of IStreamReader

Dispose pattern

A StreamWriter is an implementation of the IStreamWriter interface that generates a byte[] array.

Create IStreamWriter that writes its data to an internal byte[] buffer. It will grow as needed. Call 'GetReader' to get a IStreamReader for the written bytes. Call 'GetBytes' call to get the raw array. Only the first 'Length' bytes are valid

Returns a IStreamReader that will read the written bytes. You cannot write additional bytes to the stream after making this call.

The number of bytes written so far.

The array that holds the serialized data.

Clears any data that was previously written.

Implementation of IStreamWriter

Dispose pattern

Makespace makes at least sizeof(long) bytes available (or throws OutOfMemory)

A IOStreamStreamReader hooks a MemoryStreamReader up to an input System.IO.Stream.

Create a new IOStreamStreamReader from the given file.

Create a new IOStreamStreamReader from the given System.IO.Stream. Optionally you can specify the size of the read buffer The stream will be closed by the IOStreamStreamReader when it is closed.

close the file or underlying stream and clean up

Implementation of IStreamReader

Implementation of MemoryStreamReader

Dispose pattern

Fill the buffer, making sure at least 'minimum' byte are available to read. Throw an exception if there are not that many bytes.

A PinnedStreamReader is an IOStream reader that will pin its read buffer. This allows it it support a 'GetPointer' API efficiently. The GetPointer API lets you access data from the stream as raw byte blobs without having to copy the data.

Create a new PinnedStreamReader that gets its data from a given file. You can optionally set the size of the read buffer.

Create a new PinnedStreamReader that gets its data from a given System.IO.Stream. You can optionally set the size of the read buffer. The stream will be closed by the PinnedStreamReader when it is closed.

Clone the PinnnedStreamReader so that it reads from the same stream as this one. They will share the same System.IO.Stream, but each will lock and seek when accessing that stream so they can both safely share it.

Get a byte* pointer to the input buffer at 'Position' in the IReadStream that is at least 'length' bytes long. (thus ptr to ptr+len is valid). Note that length cannot be larger than the buffer size passed to the reader when it was constructed.

Get a byte* pointer to the input buffer at the current read position is at least 'length' bytes long. (thus ptr to ptr+len is valid). Note that length cannot be larger than the buffer size passed to the reader when it was constructed.

A IOStreamStreamWriter hooks a MemoryStreamWriter up to an output System.IO.Stream

Create a IOStreamStreamWriter that writes its data to a given file that it creates

Create a IOStreamStreamWriter that writes its data to a System.IO.Stream

Flush any written data to the underlying System.IO.Stream

Ensures the bytes in the stream are written to the stream and cleans up resources.

Access the underlying System.IO.Stream. You should avoid using this if at all possible.

Implementation of the MemoryStreamWriter interface

Implementation of the IStreamWriter interface

Implementation of the MemoryStreamWriter interface

Dispose pattern

A cheap version of List(T). The idea is to make it as cheap as if you did it 'by hand' using an array and an int which represents the logical charCount. It is a struct to avoid an extra pointer dereference, so this is really meant to be embedded in other structures.

Create a growable array with the given initial size it will grow as needed. There is also the default constructor that assumes initialSize of 0 (and does not actually allocate the array.

Fetch the element at the given index. Will throw an IndexOutOfRange exception otherwise

The number of elements in the array

Remove all elements in the array.

Add an item at the end of the array, growing as necessary.

Add all items 'items' to the end of the array, growing as necessary.

Insert 'item' directly at 'index', shifting all items >= index up. 'index' can be code:Count in which case the item is appended to the end. Larger indexes are not allowed.

Remove 'count' elements starting at 'index'

Sets the 'index' element to 'value' growing the array if necessary (filling in default values if necessary).

Gets the value at 'index'. Never fails, will return 'default' if out of range.

Returns true if there are no elements in the array.

Remove the last element added and return it. Will throw if there are no elements.

Returns the last element added Will throw if there are no elements.

Trims the size of the array so that no more than 'maxWaste' slots are wasted. Useful when you know that the array has stopped growing.

Returns true if the Growable array was initialized by the default constructor which has no capacity (and thus will cause growth on the first addition). This method allows you to lazily set the compacity of your GrowableArray by testing if it is of EmtpyCapacity, and if so set it to some useful capacity. This avoids unnecessary reallocs to get to a reasonable capacity.

A string representing the array. Only intended for debugging.

Sets 'index' to the the smallest index such that all elements with index > 'idx' are > key. If index does not match any elements a new element should always be placed AFTER index. Note that this means that index may be -1 if the new element belongs in the first position. Returns true if the return index matched exactly (success) TODO FIX NOW harmonize with List.BinarySearch

Sort the range starting at 'index' of length 'count' using 'comparision' in assending order

Sort the whole array using 'comparison' in ascending order

Executes 'func' for each element in the GrowableArray and returns a GrowableArray for the result.

Perform a linear search starting at 'startIndex'. If found return true and the index in 'index'. It is legal that 'startIndex' is greater than the charCount, in which case, the search returns false immediately. This allows a nice loop to find all items matching a pattern.

Returns the underlying array. Should not be used most of the time!

Implementation of foreach protocol

Enumerator for foreach interface

implementation of IEnumerable interface

Represents a collection of keys and values.

This collection has the similar performance characteristics as , but uses segmented lists to avoid allocations in the Large Object Heap. This implementation was based on the SegmentedDictionary implementation made for dotnet/roslyn. Original source code: https://github.com/dotnet/roslyn/blob/release/dev17.0/src/Dependencies/Collections/SegmentedDictionary%602.cs The type of the keys in the dictionary. The type of the values in the dictionary.

0-based index of next entry in chain: -1 means end of chain also encodes whether this entry _itself_ is part of the free list by changing sign and subtracting 3, so -2 means end of free list, -3 means index 0 but on free list, -4 means index 1 but on free list, etc.

Segmented list implementation, copied from Microsoft.Exchange.Collections.

The type of the list element. This class implement a list which is allocated in segments, to avoid large lists to go into LOH.

Constructs SegmentedList.

Segment size

Constructs SegmentedList.

Segment size Initial capacity

Returns the count of elements in the list.

Copy to Array

Array copy

Returns the last element on the list and removes it from it.

The last element that was on the list.

Returns true if this ICollection is read-only.

Gets or sets the given element in the list.

Element index.

Gets or sets the given element in the list.

Element index.

Necessary if the list is being used as an array since it creates the segments lazily.

true if the segment is allocated and false otherwise

Get slot of an element

Adds new element at the end of the list.

New element.

Inserts new element at the given position in the list.

Insert position. New element to insert.

Removes element at the given position in the list.

Position of the element to remove.

Performs a binary search in a sorted list.

Element to search for. Comparer to use. Non-negative position of the element if found, negative binary complement of the position of the next element if not found. The implementation was copied from CLR BinarySearch implementation.

Performs a binary search in a sorted list.

Element to search for. The lowest index in which to search. The highest index in which to search. Comparer to use. The index

Sorts the list using default comparer for elements.

Sorts the list using specified comparer for elements.

Comparer to use.

Appends a range of elements from another list.

Source list. Start index in the source list. Count of elements from the source list to append.

Appends a range of elements from another array.

Source array. Start index in the source list. Count of elements from the source list to append.

Returns the enumerator.

Copy to Array

Array copy

CopyTo copies a collection into an Array, starting at a particular index into the array.

Destination array. Destination array starting index.

Copies the contents of the collection that are within a range into an Array, starting at a particular index into the array.

Destination array. Destination array starting index. The collection index from where the copying should start. The collection index where the copying should end.

Returns the enumerator.

Clears the list (removes all elements).

Check if ICollection contains the given element.

Element to check.

CopyTo copies a collection into an Array, starting at a particular index into the array.

Destination array. Destination array starting index.

Removes the given element from this ICollection.

Element to remove.

Shifts the tail of the list to make room for a new inserted element.

Index of a new inserted element.

Shifts the tail of the list to remove the element.

Index of the removed element.

Ensures that we have enough capacity for the given number of elements.

Number of elements.

Helper method for QuickSort.

Comparer to use. Position of the first element. Position of the second element.

QuickSort implementation.

left boundary. right boundary. Comparer to use. The implementation was copied from CLR QuickSort implementation.

Enumerator over the segmented list.

Constructws the Enumerator.

List to enumerate.

Disposes the Enumerator.

Moves to the nest element in the list.

True if move successful, false if there are no more elements.

Returns the current element.

Resets the enumerator to initial state.

Returns approximate reciprocal of the divisor: ceil(2**64 / divisor).

This should only be used on 64-bit.

Performs a mod operation using the multiplier pre-computed with .

PERF: This improves performance in 64-bit scenarios at the expense of performance in 32-bit scenarios. Since we only build a single AnyCPU binary, we opt for improved performance in the 64-bit scenario.

Utility class for exception throwing for the SegmentedDictionary.