Add m-esr52 at 52.6.0

author: Matt A. Tobin <mattatobin@localhost.localdomain> 2018-02-02 04:16:08 -0500
committer: Matt A. Tobin <mattatobin@localhost.localdomain> 2018-02-02 04:16:08 -0500
commit: 5f8de423f190bbb79a62f804151bc24824fa32d8 (patch)
tree: 10027f336435511475e392454359edea8e25895d /python/pyasn1/doc/codecs.html
parent: 49ee0794b5d912db1f95dce6eb52d781dc210db5 (diff)
download: UXP-5f8de423f190bbb79a62f804151bc24824fa32d8.tar
UXP-5f8de423f190bbb79a62f804151bc24824fa32d8.tar.gz
UXP-5f8de423f190bbb79a62f804151bc24824fa32d8.tar.lz
UXP-5f8de423f190bbb79a62f804151bc24824fa32d8.tar.xz
UXP-5f8de423f190bbb79a62f804151bc24824fa32d8.zip
1 files changed, 503 insertions, 0 deletions
diff --git a/python/pyasn1/doc/codecs.html b/python/pyasn1/doc/codecs.html
new file mode 100644
index 000000000..9c2c36ed6
--- /dev/null
+++ b/python/pyasn1/doc/codecs.html
@@ -0,0 +1,503 @@
+<html>
+<title>
+PyASN1 codecs
+</title>
+<head>
+</head>
+<body>
+<center>
+<table width=60%>
+<tr>
+<td>
+<h3>
+2. PyASN1 Codecs
+</h3>
+
+<p>
+In ASN.1 context, 
+<a href=http://en.wikipedia.org/wiki/Codec>codec</a>
+is a program that transforms between concrete data structures and a stream
+of octets, suitable for transmission over the wire. This serialized form of
+data is sometimes called <i>substrate</i> or <i>essence</i>.
+</p>
+
+<p>
+In pyasn1 implementation, substrate takes shape of Python 3 bytes or 
+Python 2 string objects.
+</p>
+
+<p>
+One of the properties of a codec is its ability to cope with incomplete
+data and/or substrate what implies codec to be stateful. In other words, 
+when decoder runs out of substrate and data item being recovered is still 
+incomplete, stateful codec would suspend and complete data item recovery 
+whenever the rest of substrate becomes available. Similarly, stateful encoder
+would encode data items in multiple steps waiting for source data to
+arrive. Codec restartability is especially important when application deals
+with large volumes of data and/or runs on low RAM. For an interesting
+discussion on codecs options and design choices, refer to
+<a href=http://directory.apache.org/subprojects/asn1/>Apache ASN.1 project</a>
+.
+</p>
+
+<p>
+As of this writing, codecs implemented in pyasn1 are all stateless, mostly
+to keep the code simple.
+</p>
+
+<p>
+The pyasn1 package currently supports 
+<a href=http://en.wikipedia.org/wiki/Basic_encoding_rules>BER</a> codec and
+its variations -- 
+<a href=http://en.wikipedia.org/wiki/Canonical_encoding_rules>CER</a> and
+<a href=http://en.wikipedia.org/wiki/Distinguished_encoding_rules>DER</a>.
+More ASN.1 codecs are planned for implementation in the future.
+</p>
+
+<a name="2.1"></a>
+<h4>
+2.1 Encoders
+</h4>
+
+<p>
+Encoder is used for transforming pyasn1 value objects into substrate. Only
+pyasn1 value objects could be serialized, attempts to process pyasn1 type
+objects will cause encoder failure.
+</p>
+
+<p>
+The following code will create a pyasn1 Integer object and serialize it with
+BER encoder:
+</p>
+
+<table bgcolor="lightgray" border=0 width=100%><TR><TD>
+<pre>
+>>> from pyasn1.type import univ
+>>> from pyasn1.codec.ber import encoder
+>>> encoder.encode(univ.Integer(123456))
+b'\x02\x03\x01\xe2@'
+>>>
+</pre>
+</td></tr></table>
+
+<p>
+BER standard also defines a so-called <i>indefinite length</i> encoding form
+which makes large data items processing more memory efficient. It is mostly
+useful when encoder does not have the whole value all at once and the
+length of the value can not be determined at the beginning of encoding.
+</p>
+
+<p>
+<i>Constructed encoding</i> is another feature of BER closely related to the
+indefinite length form. In essence, a large scalar value (such as ASN.1
+character BitString type) could be chopped into smaller chunks by encoder
+and transmitted incrementally to limit memory consumption. Unlike indefinite
+length case, the length of the whole value must be known in advance when
+using constructed, definite length encoding form.
+</p>
+
+<p>
+Since pyasn1 codecs are not restartable, pyasn1 encoder may only encode data
+item all at once. However, even in this case, generating indefinite length 
+encoding may help a low-memory receiver, running a restartable decoder,
+to process a large data item.
+</p>
+
+<table bgcolor="lightgray" border=0 width=100%><TR><TD>
+<pre>
+>>> from pyasn1.type import univ
+>>> from pyasn1.codec.ber import encoder
+>>> encoder.encode(
+...   univ.OctetString('The quick brown fox jumps over the lazy dog'),
+...   defMode=False,
+...   maxChunkSize=8
+... )
+b'$\x80\x04\x08The quic\x04\x08k brown \x04\x08fox jump\x04\x08s over \
+t\x04\x08he lazy \x04\x03dog\x00\x00'
+>>>
+>>> encoder.encode(
+...   univ.OctetString('The quick brown fox jumps over the lazy dog'),
+...   maxChunkSize=8
+... )
+b'$7\x04\x08The quic\x04\x08k brown \x04\x08fox jump\x04\x08s over \
+t\x04\x08he lazy \x04\x03dog'
+</pre>
+</td></tr></table>
+
+<p>
+The <b>defMode</b> encoder parameter disables definite length encoding mode,
+while the optional <b>maxChunkSize</b> parameter specifies desired
+substrate chunk size that influences memory requirements at the decoder's end.
+</p>
+
+<p>
+To use CER or DER encoders one needs to explicitly import and call them - the
+APIs are all compatible.
+</p>
+
+<table bgcolor="lightgray" border=0 width=100%><TR><TD>
+<pre>
+>>> from pyasn1.type import univ
+>>> from pyasn1.codec.ber import encoder as ber_encoder
+>>> from pyasn1.codec.cer import encoder as cer_encoder
+>>> from pyasn1.codec.der import encoder as der_encoder
+>>> ber_encoder.encode(univ.Boolean(True))
+b'\x01\x01\x01'
+>>> cer_encoder.encode(univ.Boolean(True))
+b'\x01\x01\xff'
+>>> der_encoder.encode(univ.Boolean(True))
+b'\x01\x01\xff'
+>>>
+</pre>
+</td></tr></table>
+
+<a name="2.2"></a>
+<h4>
+2.2 Decoders
+</h4>
+
+<p>
+In the process of decoding, pyasn1 value objects are created and linked to
+each other, based on the information containted in the substrate. Thus,
+the original pyasn1 value object(s) are recovered.
+</p>
+
+<table bgcolor="lightgray" border=0 width=100%><TR><TD>
+<pre>
+>>> from pyasn1.type import univ
+>>> from pyasn1.codec.ber import encoder, decoder
+>>> substrate = encoder.encode(univ.Boolean(True))
+>>> decoder.decode(substrate)
+(Boolean('True(1)'), b'')
+>>>
+</pre>
+</td></tr></table>
+
+<p>
+Commenting on the code snippet above, pyasn1 decoder accepts substrate
+as an argument and returns a tuple of pyasn1 value object (possibly
+a top-level one in case of constructed object) and unprocessed part
+of input substrate.
+</p>
+
+<p>
+All pyasn1 decoders can handle both definite and indefinite length
+encoding modes automatically, explicit switching into one mode
+to another is not required.
+</p>
+
+<table bgcolor="lightgray" border=0 width=100%><TR><TD>
+<pre>
+>>> from pyasn1.type import univ
+>>> from pyasn1.codec.ber import encoder, decoder
+>>> substrate = encoder.encode(
+...   univ.OctetString('The quick brown fox jumps over the lazy dog'),
+...   defMode=False,
+...   maxChunkSize=8
+... )
+>>> decoder.decode(substrate)
+(OctetString(b'The quick brown fox jumps over the lazy dog'), b'')
+>>>
+</pre>
+</td></tr></table>
+
+<p>
+Speaking of BER/CER/DER encoding, in many situations substrate may not contain
+all necessary information needed for complete and accurate ASN.1 values
+recovery. The most obvious cases include implicitly tagged ASN.1 types
+and constrained types.
+</p>
+
+<p>
+As discussed earlier in this handbook, when an ASN.1 type is implicitly
+tagged, previous outermost tag is lost and never appears in substrate.
+If it is the base tag that gets lost, decoder is unable to pick type-specific
+value decoder at its table of built-in types, and therefore recover
+the value part, based only on the information contained in substrate. The
+approach taken by pyasn1 decoder is to use a prototype pyasn1 type object (or
+a set of them) to <i>guide</i> the decoding process by matching [possibly
+incomplete] tags recovered from substrate with those found in prototype pyasn1
+type objects (also called pyasn1 specification object further in this paper).
+</p>
+
+<table bgcolor="lightgray" border=0 width=100%><TR><TD>
+<pre>
+>>> from pyasn1.codec.ber import decoder
+>>> decoder.decode(b'\x02\x01\x0c', asn1Spec=univ.Integer())
+Integer(12), b''
+>>>
+</pre>
+</td></tr></table>
+
+<p>
+Decoder would neither modify pyasn1 specification object nor use
+its current values (if it's a pyasn1 value object), but rather use it as
+a hint for choosing proper decoder and as a pattern for creating new objects:
+</p>
+
+<table bgcolor="lightgray" border=0 width=100%><TR><TD>
+<pre>
+>>> from pyasn1.type import univ, tag
+>>> from pyasn1.codec.ber import encoder, decoder
+>>> i = univ.Integer(12345).subtype(
+...   implicitTag=tag.Tag(tag.tagClassContext, tag.tagFormatSimple, 40)
+... )
+>>> substrate = encoder.encode(i)
+>>> substrate
+b'\x9f(\x0209'
+>>> decoder.decode(substrate)
+Traceback (most recent call last):
+...
+pyasn1.error.PyAsn1Error: 
+   TagSet(Tag(tagClass=128, tagFormat=0, tagId=40)) not in asn1Spec
+>>> decoder.decode(substrate, asn1Spec=i)
+(Integer(12345), b'')
+>>>
+</pre>
+</td></tr></table>
+
+<p>
+Notice in the example above, that an attempt to run decoder without passing
+pyasn1 specification object fails because recovered tag does not belong
+to any of the built-in types.
+</p>
+
+<p>
+Another important feature of guided decoder operation is the use of
+values constraints possibly present in pyasn1 specification object.
+To explain this, we will decode a random integer object into generic Integer
+and the constrained one.
+</p>
+
+<table bgcolor="lightgray" border=0 width=100%><TR><TD>
+<pre>
+>>> from pyasn1.type import univ, constraint
+>>> from pyasn1.codec.ber import encoder, decoder
+>>> class DialDigit(univ.Integer):
+...   subtypeSpec = constraint.ValueRangeConstraint(0,9)
+>>> substrate = encoder.encode(univ.Integer(13))
+>>> decoder.decode(substrate)
+(Integer(13), b'')
+>>> decoder.decode(substrate, asn1Spec=DialDigit())
+Traceback (most recent call last):
+...
+pyasn1.type.error.ValueConstraintError:
+  ValueRangeConstraint(0, 9) failed at: 13
+>>> 
+</pre>
+</td></tr></table>
+
+<p>
+Similarily to encoders, to use CER or DER decoders application has to
+explicitly import and call them - all APIs are compatible.
+</p>
+
+<table bgcolor="lightgray" border=0 width=100%><TR><TD>
+<pre>
+>>> from pyasn1.type import univ
+>>> from pyasn1.codec.ber import encoder as ber_encoder
+>>> substrate = ber_encoder.encode(univ.OctetString('http://pyasn1.sf.net'))
+>>>
+>>> from pyasn1.codec.ber import decoder as ber_decoder
+>>> from pyasn1.codec.cer import decoder as cer_decoder
+>>> from pyasn1.codec.der import decoder as der_decoder
+>>> 
+>>> ber_decoder.decode(substrate)
+(OctetString(b'http://pyasn1.sf.net'), b'')
+>>> cer_decoder.decode(substrate)
+(OctetString(b'http://pyasn1.sf.net'), b'')
+>>> der_decoder.decode(substrate)
+(OctetString(b'http://pyasn1.sf.net'), b'')
+>>> 
+</pre>
+</td></tr></table>
+
+<a name="2.2.1"></a>
+<h4>
+2.2.1 Decoding untagged types
+</h4>
+
+<p>
+It has already been mentioned, that ASN.1 has two "special case" types:
+CHOICE and ANY. They are different from other types in part of 
+tagging - unless these two are additionally tagged, neither of them will
+have their own tag. Therefore these types become invisible in substrate
+and can not be recovered without passing pyasn1 specification object to
+decoder.
+</p>
+
+<p>
+To explain the issue, we will first prepare a Choice object to deal with:
+</p>
+
+<table bgcolor="lightgray" border=0 width=100%><TR><TD>
+<pre>
+>>> from pyasn1.type import univ, namedtype
+>>> class CodeOrMessage(univ.Choice):
+...   componentType = namedtype.NamedTypes(
+...     namedtype.NamedType('code', univ.Integer()),
+...     namedtype.NamedType('message', univ.OctetString())
+...   )
+>>>
+>>> codeOrMessage = CodeOrMessage()
+>>> codeOrMessage.setComponentByName('message', 'my string value')
+>>> print(codeOrMessage.prettyPrint())
+CodeOrMessage:
+ message=b'my string value'
+>>>
+</pre>
+</td></tr></table>
+
+<p>
+Let's now encode this Choice object and then decode its substrate
+with and without pyasn1 specification object:
+</p>
+
+<table bgcolor="lightgray" border=0 width=100%><TR><TD>
+<pre>
+>>> from pyasn1.codec.ber import encoder, decoder
+>>> substrate = encoder.encode(codeOrMessage)
+>>> substrate
+b'\x04\x0fmy string value'
+>>> encoder.encode(univ.OctetString('my string value'))
+b'\x04\x0fmy string value'
+>>>
+>>> decoder.decode(substrate)
+(OctetString(b'my string value'), b'')
+>>> codeOrMessage, substrate = decoder.decode(substrate, asn1Spec=CodeOrMessage())
+>>> print(codeOrMessage.prettyPrint())
+CodeOrMessage:
+ message=b'my string value'
+>>>
+</pre>
+</td></tr></table>
+
+<p>
+First thing to notice in the listing above is that the substrate produced
+for our Choice value object is equivalent to the substrate for an OctetString
+object initialized to the same value. In other words, any information about
+the Choice component is absent in encoding.
+</p>
+
+<p>
+Sure enough, that kind of substrate will decode into an OctetString object,
+unless original Choice type object is passed to decoder to guide the decoding
+process.
+</p>
+
+<p>
+Similarily untagged ANY type behaves differently on decoding phase - when
+decoder bumps into an Any object in pyasn1 specification, it stops decoding
+and puts all the substrate into a new Any value object in form of an octet
+string. Concerned application could then re-run decoder with an additional,
+more exact pyasn1 specification object to recover the contents of Any
+object.
+</p>
+
+<p>
+As it was mentioned elsewhere in this paper, Any type allows for incomplete
+or changing ASN.1 specification to be handled gracefully by decoder and
+applications.
+</p>
+
+<p>
+To illustrate the working of Any type, we'll have to make the stage
+by encoding a pyasn1 object and then putting its substrate into an any
+object.
+</p>
+
+<table bgcolor="lightgray" border=0 width=100%><TR><TD>
+<pre>
+>>> from pyasn1.type import univ
+>>> from pyasn1.codec.ber import encoder, decoder
+>>> innerSubstrate = encoder.encode(univ.Integer(1234))
+>>> innerSubstrate
+b'\x02\x02\x04\xd2'
+>>> any = univ.Any(innerSubstrate)
+>>> any
+Any(b'\x02\x02\x04\xd2')
+>>> substrate = encoder.encode(any)
+>>> substrate
+b'\x02\x02\x04\xd2'
+>>>
+</pre>
+</td></tr></table>
+
+<p>
+As with Choice type encoding, there is no traces of Any type in substrate.
+Obviously, the substrate we are dealing with, will decode into the inner
+[Integer] component, unless pyasn1 specification is given to guide the 
+decoder. Continuing previous code:
+</p>
+
+<table bgcolor="lightgray" border=0 width=100%><TR><TD>
+<pre>
+>>> from pyasn1.type import univ
+>>> from pyasn1.codec.ber import encoder, decoder
+
+>>> decoder.decode(substrate)
+(Integer(1234), b'')
+>>> any, substrate = decoder.decode(substrate, asn1Spec=univ.Any())
+>>> any
+Any(b'\x02\x02\x04\xd2')
+>>> decoder.decode(str(any))
+(Integer(1234), b'')
+>>>
+</pre>
+</td></tr></table>
+
+<p>
+Both CHOICE and ANY types are widely used in practice. Reader is welcome to
+take a look at 
+<a href=http://www.cs.auckland.ac.nz/~pgut001/pubs/x509guide.txt>
+ASN.1 specifications of X.509 applications</a> for more information.
+</p>
+
+<a name="2.2.2"></a>
+<h4>
+2.2.2 Ignoring unknown types
+</h4>
+
+<p>
+When dealing with a loosely specified ASN.1 structure, the receiving
+end may not be aware of some types present in the substrate. It may be
+convenient then to turn decoder into a recovery mode. Whilst there, decoder
+will not bail out when hit an unknown tag but rather treat it as an Any
+type.
+</p>
+
+<table bgcolor="lightgray" border=0 width=100%><TR><TD>
+<pre>
+>>> from pyasn1.type import univ, tag
+>>> from pyasn1.codec.ber import encoder, decoder
+>>> taggedInt = univ.Integer(12345).subtype(
+...   implicitTag=tag.Tag(tag.tagClassContext, tag.tagFormatSimple, 40)
+... )
+>>> substrate = encoder.encode(taggedInt)
+>>> decoder.decode(substrate)
+Traceback (most recent call last):
+...
+pyasn1.error.PyAsn1Error: TagSet(Tag(tagClass=128, tagFormat=0, tagId=40)) not in asn1Spec
+>>>
+>>> decoder.decode.defaultErrorState = decoder.stDumpRawValue
+>>> decoder.decode(substrate)
+(Any(b'\x9f(\x0209'), '')
+>>>
+</pre>
+</td></tr></table>
+
+<p>
+It's also possible to configure a custom decoder, to handle unknown tags
+found in substrate. This can be done by means of <b>defaultRawDecoder</b>
+attribute holding a reference to type decoder object. Refer to the source
+for API details.
+</p>
+
+<hr>
+
+</td>
+</tr>
+</table>
+</center>
+</body>
+</html>
author	Matt A. Tobin <mattatobin@localhost.localdomain>	2018-02-02 04:16:08 -0500
committer	Matt A. Tobin <mattatobin@localhost.localdomain>	2018-02-02 04:16:08 -0500
commit	5f8de423f190bbb79a62f804151bc24824fa32d8 (patch)
tree	10027f336435511475e392454359edea8e25895d /python/pyasn1/doc/codecs.html
parent	49ee0794b5d912db1f95dce6eb52d781dc210db5 (diff)
download	UXP-5f8de423f190bbb79a62f804151bc24824fa32d8.tar UXP-5f8de423f190bbb79a62f804151bc24824fa32d8.tar.gz UXP-5f8de423f190bbb79a62f804151bc24824fa32d8.tar.lz UXP-5f8de423f190bbb79a62f804151bc24824fa32d8.tar.xz UXP-5f8de423f190bbb79a62f804151bc24824fa32d8.zip