Better way to read a serial data stream?

[PicDude]

Hi all,

I intend to read a proprietary serial data stream from another product into a spare PortA pin on a 16F pic.  Currently 8Mhz internal osc, but the PCB is setup for a 3-pin resonator (20Mhz or other) resonator if necessary.

Essentially there is a relatively long idle period, followed by over 70 sequential data bits which are around 400ms long each, and at approx 1% timing error from the sender.  This becomes fun because I don't have interrupt-on-change on this pin, and trying to set a constant time for each bit will be difficult because and error will accumulate over 70 bits.

So my plan is to generate Timer 2 interrupts every 50 to 100 ms looking for the idle period, then end-of-idle transition, adjust the timer (PR2) to poll in the middle as appropriate to capture the bit value, then adjust PR2 again to end a few percent before the expected transition, and then poll in a loop to accurately determine the edge/transition that marks the end of the bit (the beginning of the bit).  The "few percent" will allow for all sources of timing errors from the sender, receiver (pic) clock, etc.

Now the question -- since the pic is doing other things, the timer interrupt that should be triggered a few percent before the end of the bit may get delayed, so the few percent may change to 5 percent or more, and I could be waiting for maybe over 100 clock cycles before returning from that interrupt.  I hate doing that, and always try to keep my interrupt processing as short as possible.  Is there another/better way to do this?

Cheers,
-Neil.

[PicDude]

Oops, this should be 400 us (*micro*seconds) long each.

PicDude wrote:

... over 70 sequential data bits which are around 400ms long each...

[Richard Prosser]

2009/8/18 PicDude <picdude2@...>:

>
> Oops, this should be 400 us (*micro*seconds) long each.
>
>
>
> PicDude wrote:
>>
>> ... over 70 sequential data bits which are around 400ms long each...
>>
>
> --
> View this message in context: http://www.nabble.com/Better-way-to-read-a-serial-data-stream--tp25018655p25018667.html
> Sent from the PIC - [PIC] mailing list archive at Nabble.com.
>

Picdude.
I think you need to organise a system in which the errors don't
accumulate. I used  a method (many years ago) that maintained bit
synch  that may be useful. Trouble is it's a bit hard to explain.

Basically you sample twice per  period, once at the transition and
once at the anticipated midpoint. If the midpoint reading agrees with
the transition reading but the bit has changed from the previous bit
(midpoint), then you assume that you have sampled late and your clock
is a bit slow. Similarly, if the bit has changed, and so has the
reading since the transition, then you figure your local clock is fast
and adjust it slightly accordingly. Providing the adjustment is small,
you can retain lock in quite noisy environments - we were using a
radio link.

It looks a bit complicated but actually coded into quite a compact
block of assembler. ('6805 in the case in point)

Richard P
--
http://www.piclist.com PIC/SX FAQ & list archive
View/change your membership options at
http://mailman.mit.edu/mailman/listinfo/piclist

[Michael Rigby-Jones-2]

> -----Original Message-----
> From: piclist-bounces@... [mailto:piclist-bounces@...] On
Behalf

> Of PicDude
> Sent: 18 August 2009 06:20
> To: piclist@...
> Subject: [PIC] Better way to read a serial data stream?
>
>
> Hi all,
>
> I intend to read a proprietary serial data stream from another product
> into
> a spare PortA pin on a 16F pic.  Currently 8Mhz internal osc, but the

PCB
> is
> setup for a 3-pin resonator (20Mhz or other) resonator if necessary.
>
> Essentially there is a relatively long idle period, followed by over
70
> sequential data bits which are around 400ms long each, and at approx
1%
> timing error from the sender.  This becomes fun because I don't have
> interrupt-on-change on this pin,

Which pin are you using and which device?

=======================================================================
This e-mail is intended for the person it is addressed to only. The
information contained in it may be confidential and/or protected by
law. If you are not the intended recipient of this message, you must
not make any use of this information, or copy or show it to any
person. Please contact us immediately to tell us that you have
received this e-mail, and return the original to us. Any use,
forwarding, printing or copying of this message is strictly prohibited.
No part of this message can be considered a request for goods or
services.
=======================================================================

--
http://www.piclist.com PIC/SX FAQ & list archive
View/change your membership options at
http://mailman.mit.edu/mailman/listinfo/piclist

[Olin Lathrop]

PicDude wrote:

> I intend to read a proprietary serial data stream from another
> product into
> a spare PortA pin on a 16F pic.  Currently 8Mhz internal osc, but the
> PCB is
> setup for a 3-pin resonator (20Mhz or other) resonator if necessary.
>
> Essentially there is a relatively long idle period, followed by over
> 70
> sequential data bits which are around 400ms long each, and at approx
> 1%
> timing error from the sender.

Obviously this can't work.  Just from the sender's clock error alone you are
off by .7 bits in the middle of the last bit, with .5 bit time error being
the guaranteed to fail limit.  Any error in the receiving process adds to
this.  Either you got the specs wrong or some idiot designed a protocol that
can't be correctly received regardless of how much cleverness is applied in
the receiver.  If this is really NRZ encoding and the sender could send all
bits the same value, then you're screwed.

You can resync your clock on bit transitions, assuming these happen once
between bits and the bits are the same length.  Even though bit value
changes may not be guarantee within a particular time, you can still make
use of them when they do occur, and most likely they will occur often enough
in practise.

The easiest thing is to set up periodic interrupt and sample the input
signal there.  Fortunately your bit rate is very slow, so this interrupt
could every 10mS or more.  Even something as slow as 10mS still gives you 40
samples/bit.

Obviosly you watch for the start bit and start timing there, but you also
watch for transitions on the line and reset the clock to the middle between
two bits when you find one.  That resets any accumulated clock drift between
the sender and receiver on every bit value change.  The problem is that
these are not guaranteed to occur unless you haven't told us everything.

For extra credit, you not only reset the clock on input line transitions,
but also adjust your conception of a bit time a little.  After a few
messages, your PIC will have been trained on that sender, and subsequent
messages will work better, even with long sequences of the same bit value.

A easy way to do this is to keep a number that indicates how many samples
there are per bit.  This trick is to make this fixed point and keep some
fraction bits, like 8.  This allows making small changes in your assumed bit
period when edges are found.  I would adjust the bit period slowly, probably
by low pass filtering any new value into the in-use value with a couple of
poles and a low filter fraction.  The error between the sender's clock and
your clock shouldn't change much, and should change slowly when it does.


********************************************************************
Embed Inc, Littleton Massachusetts, http://www.embedinc.com/products
[phone].  Gold level PIC consultants since 2000.
--
http://www.piclist.com PIC/SX FAQ & list archive
View/change your membership options at
http://mailman.mit.edu/mailman/listinfo/piclist

[William "Chops" Westfield]

On Aug 18, 2009, at 5:22 AM, Olin Lathrop wrote:

>> Essentially there is a relatively long idle period, followed by  
>> over 70 sequential data bits which are around 400ms long each, and  
>> at approx 1% timing error from the sender.
>
> Obviously this can't work.

I agree with Olin.  This is where schemes other than NRZ come in (even  
just stop and start bits in async serial), and why they're needed...

BillW


--
http://www.piclist.com PIC/SX FAQ & list archive
View/change your membership options at
http://mailman.mit.edu/mailman/listinfo/piclist

[PicDude]

Richard Prosser wrote:

2009/8/18 PicDude <[e-mail]>:
>
> Oops, this should be 400 us (*micro*seconds) long each.
>
>
>
> PicDude wrote:
>>
>> ... over 70 sequential data bits which are around 400ms long each...
>>
>
> --
> View this message in context: http://www.nabble.com/Better-way-to-read-a-serial-data-stream--tp25018655p25018667.html
> Sent from the PIC - [PIC] mailing list archive at Nabble.com.
>


Picdude.
I think you need to organise a system in which the errors don't
accumulate. I used  a method (many years ago) that maintained bit
synch  that may be useful. Trouble is it's a bit hard to explain.

Basically you sample twice per  period, once at the transition and
once at the anticipated midpoint. If the midpoint reading agrees with
the transition reading but the bit has changed from the previous bit
(midpoint), then you assume that you have sampled late and your clock
is a bit slow. Similarly, if the bit has changed, and so has the
reading since the transition, then you figure your local clock is fast
and adjust it slightly accordingly. Providing the adjustment is small,
you can retain lock in quite noisy environments - we were using a
radio link.

It looks a bit complicated but actually coded into quite a compact
block of assembler. ('6805 in the case in point)

Richard P
--
http://www.piclist.com PIC/SX FAQ & list archive
View/change your membership options at
http://mailman.mit.edu/mailman/listinfo/piclist

Yes I do (need to keep the errors from accumulating), hence the part about polling in a busy loop to wait until the end-of-bit transition.  But I was wondering if there's a better way, rather than looping idly within the interrupt service routine.

Let me clarify a few things, which I think will make a difference -- the bits start and end with a L-->H transition.  The H-->L transition point between the start and end points of each bit varies, so the duty-cycle determines the bit value.  This is not NRZ.  Yes, bits are the same length.

I'm using RA1 on a 16F883 currently.

[Dave Tweed]

PicDude wrote:
> Let me clarify a few things, which I think will make a difference -- the
> bits start and end with a L-->H transition. The H-->L transition point
> between the start and end points of each bit varies, so the duty-cycle
> determines the bit value. This is not NRZ. Yes, bits are the same length.

In that case, you have a self-clocking code, which makes things a LOT
easier.

You just need to poll the data line with a period that is less than (with a
comfortable margin) the minimum interval between any pair of transitions.
For example, if the signal is 1/3 high, 2/3 low for a 0 and vice versa for
a 1, then sample at 4x the bit rate, or 100 us. The timing margin you
select should include any expected peak-to-peak jitter in the sampling that
might be due to interrupt latencies, etc.

To decode the data, simply look for groups samples consisting of contiguous
1s followed by contiguous 0s, and make your bit decision based on the
relative counts in each group. This can even be done in the ISR itself,
with little overhead.

-- Dave Tweed
--
http://www.piclist.com PIC/SX FAQ & list archive
View/change your membership options at
http://mailman.mit.edu/mailman/listinfo/piclist

[Richard Prosser]

2009/8/20 PicDude <picdude2@...>:

>
>
> Richard Prosser wrote:
>>
>> 2009/8/18 PicDude <picdude2@...>:
>>>
>>> Oops, this should be 400 us (*micro*seconds) long each.
>>>
>>>
>>>
>>> PicDude wrote:
>>>>
>>>> ... over 70 sequential data bits which are around 400ms long each...
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/Better-way-to-read-a-serial-data-stream--tp25018655p25018667.html
>>> Sent from the PIC - [PIC] mailing list archive at Nabble.com.
>>>
>>
>>
>> Picdude.
>> I think you need to organise a system in which the errors don't
>> accumulate. I used  a method (many years ago) that maintained bit
>> synch  that may be useful. Trouble is it's a bit hard to explain.
>>
>> Basically you sample twice per  period, once at the transition and
>> once at the anticipated midpoint. If the midpoint reading agrees with
>> the transition reading but the bit has changed from the previous bit
>> (midpoint), then you assume that you have sampled late and your clock
>> is a bit slow. Similarly, if the bit has changed, and so has the
>> reading since the transition, then you figure your local clock is fast
>> and adjust it slightly accordingly. Providing the adjustment is small,
>> you can retain lock in quite noisy environments - we were using a
>> radio link.
>>
>> It looks a bit complicated but actually coded into quite a compact
>> block of assembler. ('6805 in the case in point)
>>
>> Richard P
>> --
>> http://www.piclist.com PIC/SX FAQ & list archive
>> View/change your membership options at
>> http://mailman.mit.edu/mailman/listinfo/piclist
>>
>>
>
>
> Yes I do (need to keep the errors from accumulating), hence the part about
> polling in a busy loop to wait until the end-of-bit transition.  But I was
> wondering if there's a better way, rather than looping idly within the
> interrupt service routine.
>
> Let me clarify a few things, which I think will make a difference -- the
> bits start and end with a L-->H transition.  The H-->L transition point
> between the start and end points of each bit varies, so the duty-cycle
> determines the bit value.  This is not NRZ.  Yes, bits are the same length.
>
> I'm using RA1 on a 16F883 currently.
>
>

With the sheme  suggested you don't have to loop idly, you just set
the interrupt to go off at twice the nominal bit frequency. Then
adjust the frequency slightly according to your results. Since you're
only talking about a 1% error, the adjustment only needs to be minor
and latency variations won't have a significant overall effect.

Richard P

--
http://www.piclist.com PIC/SX FAQ & list archive
View/change your membership options at
http://mailman.mit.edu/mailman/listinfo/piclist

[PicDude]

Okay, that seems like a surprisingly simple solution to this problem.  I'm evaluating on paper for now, and can't see it working well with 4x the bit rate, but 5x seems to work well... so far.  This is assuming minimal error for now, but I'll try factoring in a few percent error to see how 5x would handle it.  What's nice is that the ISR processing for this is indeed very minimal, so a faster sample rate is fine.

FWIW, I've been able to re-work some of my other ISR code (mostly to multiplex LED displays) to reduce interrupt latency by breaking it up into a couple shorter calls.

Thanks,
-Neil.

Dave Tweed wrote:

PicDude wrote:
> Let me clarify a few things, which I think will make a difference -- the
> bits start and end with a L-->H transition. The H-->L transition point
> between the start and end points of each bit varies, so the duty-cycle
> determines the bit value. This is not NRZ. Yes, bits are the same length.

In that case, you have a self-clocking code, which makes things a LOT
easier.

You just need to poll the data line with a period that is less than (with a
comfortable margin) the minimum interval between any pair of transitions.
For example, if the signal is 1/3 high, 2/3 low for a 0 and vice versa for
a 1, then sample at 4x the bit rate, or 100 us. The timing margin you
select should include any expected peak-to-peak jitter in the sampling that
might be due to interrupt latencies, etc.

To decode the data, simply look for groups samples consisting of contiguous
1s followed by contiguous 0s, and make your bit decision based on the
relative counts in each group. This can even be done in the ISR itself,
with little overhead.

-- Dave Tweed
--
http://www.piclist.com PIC/SX FAQ & list archive
View/change your membership options at
http://mailman.mit.edu/mailman/listinfo/piclist

[Dave Tweed]

PicDude wrote:
> Okay, that seems like a surprisingly simple solution to this problem.
> I'm evaluating on paper for now, and can't see it working well with 4x
> the bit rate, but 5x seems to work well... so far. This is assuming
> minimal error for now, but I'll try factoring in a few percent error
> to see how 5x would handle it. What's nice is that the ISR processing
> for this is indeed very minimal, so a faster sample rate is fine.

Like I said, the interval depends on the exact characteristics of the data
you're trying to decode.

Actully, I misspoke. Suppose the data is high for 40% of the bit interval
for a zero, and 60% for a one. In that case, you'd need to sample with a
period that is less than -- with some margin -- 20% of the bit interval in
order to reliably resolve the difference.

-- Dave Tweed
--
http://www.piclist.com PIC/SX FAQ & list archive
View/change your membership options at
http://mailman.mit.edu/mailman/listinfo/piclist

[Bob Axtell]

I would drop the internal RC clock and use a ceramic resonator or
crystal instead. Otherwise you will be driven to despair by timing
errors,

I'd sample at least 4x per bit if possible then nudge the centerline
slightly. Timer2 and PR2 will work, I've done it. With a good clock.

--Bob

On Fri, Aug 21, 2009 at 11:12 AM, Dave Tweed<pic@...> wrote:

> PicDude wrote:
>> Okay, that seems like a surprisingly simple solution to this problem.
>> I'm evaluating on paper for now, and can't see it working well with 4x
>> the bit rate, but 5x seems to work well... so far. This is assuming
>> minimal error for now, but I'll try factoring in a few percent error
>> to see how 5x would handle it. What's nice is that the ISR processing
>> for this is indeed very minimal, so a faster sample rate is fine.
>
> Like I said, the interval depends on the exact characteristics of the data
> you're trying to decode.
>
> Actully, I misspoke. Suppose the data is high for 40% of the bit interval
> for a zero, and 60% for a one. In that case, you'd need to sample with a
> period that is less than -- with some margin -- 20% of the bit interval in
> order to reliably resolve the difference.
>
> -- Dave Tweed
> --
> http://www.piclist.com PIC/SX FAQ & list archive
> View/change your membership options at
> http://mailman.mit.edu/mailman/listinfo/piclist
>

--
http://www.piclist.com PIC/SX FAQ & list archive
View/change your membership options at
http://mailman.mit.edu/mailman/listinfo/piclist

[Dave Tweed]

Bob Axtell wrote:

> Dave Tweed<pic@...> wrote:
> > PicDude wrote:
> > > Okay, that seems like a surprisingly simple solution to this problem.
> > > I'm evaluating on paper for now, and can't see it working well with
> > > 4x the bit rate, but 5x seems to work well... so far. This is assuming
> > > minimal error for now, but I'll try factoring in a few percent error
> > > to see how 5x would handle it. What's nice is that the ISR processing
> > > for this is indeed very minimal, so a faster sample rate is fine.
> >
> > Like I said, the interval depends on the exact characteristics of the
> > data you're trying to decode.
> >
> > Actully, I misspoke. Suppose the data is high for 40% of the bit
> > interval for a zero, and 60% for a one. In that case, you'd need to
> > sample with a period that is less than -- with some margin -- 20% of
> > the bit interval in order to reliably resolve the difference.
>
> I would drop the internal RC clock and use a ceramic resonator or
> crystal instead. Otherwise you will be driven to despair by timing
> errors,
>
> I'd sample at least 4x per bit if possible then nudge the centerline
> slightly. Timer2 and PR2 will work, I've done it. With a good clock.

I think you missed the point of the scheme I'm outlining. It doesn't
require the sampling clock to be synchronized with the data at all.
The only requirement is that the sampling period be less than the time
intervals you're trying to resolve, with sufficient margin based on
things like CPU clock accuracy and sampling jitter.

-- Dave Tweed
--
http://www.piclist.com PIC/SX FAQ & list archive
View/change your membership options at
http://mailman.mit.edu/mailman/listinfo/piclist

[PicDude]

Hmmm... no, I wasn't thinking that (that it has to be synchronized).  FWIW, I got this working, but I get occasional errors if the sampling period is any higher than 1/9th the period of the data.  I suspect that most of this is due to latency from other interrupt routines in the code.  It's not a problem for now, as there's no other significant processing in the code, but I'm cleaning up (and breaking up) some of that other code now.  Thanks.

Cheers,
-Neil.

Dave Tweed wrote:

I think you missed the point of the scheme I'm outlining. It doesn't
require the sampling clock to be synchronized with the data at all.
The only requirement is that the sampling period be less than the time
intervals you're trying to resolve, with sufficient margin based on
things like CPU clock accuracy and sampling jitter.

-- Dave Tweed

[Alan B. Pearce-2]

>So my plan is to generate Timer 2 interrupts every 50 to 100 ms
>looking for the idle period, then end-of-idle transition, adjust
>the timer (PR2) to poll in the middle as appropriate to capture
>the bit value, then adjust PR2 again to end a few percent before
>the expected transition, and then poll in a loop to accurately
>determine the edge/transition that marks the end of the bit (the
>beginning of the bit).  The "few percent" will allow for all
>sources of timing errors from the sender, receiver (pic) clock, etc.
>
>Now the question -- since the pic is doing other things, the timer
>interrupt that should be triggered a few percent before the end of
>the bit may get delayed, so the few percent may change to 5 percent
>or more, and I could be waiting for maybe over 100 clock cycles
>before returning from that interrupt.  I hate doing that, and always
>try to keep my interrupt processing as short as possible.
>Is there another/better way to do this?

Right, sorry to be so late replying, but I have been away for 9 weeks
looking after my ailing father.

My response is to refer you to the M6800 Applications manual that was
printed around the late 1970s. Someone posted a link here to a downloadable
PDF copy when I mentioned this some time in the past, so check the archives.

My reason for mentioning it is that they do a scheme something like you are
suggesting, to read a barcode wand for UPC codes, which is hand scanned
across the barcode, so has to deal with variable speed scans, as well as
widely different scan rates. Should give you some good ideas on what you
need to do to get good results every time.

--
http://www.piclist.com PIC/SX FAQ & list archive
View/change your membership options at
http://mailman.mit.edu/mailman/listinfo/piclist