Frame rate dropped then increased gradually when using aravis

Using Aravis
1

Has anyone seen a camera dropped the frame during certain time every day? For example, when we set the frame rate to 20, the camera was streaming at 20 fps most of the time, but started to drop around middle night. It took the camera 2-3 hours to drop to 14 fps then increased to 20 fps after another 2-3 hours. The pipeline we are using is:

“pipelineTemplate”: “aravissrc gain-auto=false exposure-auto=true ! capsfilter caps=“video/x-bayer,width=1280,height=1024,format=rggb,framerate=20/1” ! bayer2rgb ! videoconvert ! appsink”,

2

Hi @coyun

This sounds like a load issue on your machine. Check every cron jobs that can start around midnight, or external access that could happen if your machine if offering some services, paying special attention to network use if your camera is a GigEVision one.

Emmanuel.

3

Thanks @Emmanuel. Another behavior we saw was that the same camera had cosine shape frame rate before we changed to the new pipeline above. The cosign shape reached the lowest frame rate 0 every 22 hours and reached the highest frame rate 8 fps every 22 hours. The pipeline we were using before:

“pipelineTemplate”: “aravissrc gain-auto=false exposure-auto=true ! capsfilter caps=“video/x-raw,width=1280,height=1024,format=YUY2,framerate=20/1” ! videoconvert ! appsink”,

4

Yes, our camera is GigEVision. I checked network traffic but didn’t see any additional network from cron jobs.

5

Is it getting darker at midnight? You are using automatic exposure. If you need to expose longer than the time of each frame its natural that frame rate drops.
You might need to increase camera gain.

6

It happened at local day time, and sometimes at midnight as well recently. One thing I noticed was that the network packets dropped 5% from camera to docker container, and only dropped <0.5% from camera to host. It sounds like the traffic from camera to host has enough network buffer size, but the traffic from host to camera does not.

7

This is a Balser camera. I didn’t see correlation between the network packet drop and the frame rate. Maybe the GevSCPD and GevTimestampTickFrequency could affect the frame rate?

8

Camera gainAuto is off. I didn’t see an option to increase the gain.

9

GevTimestampTickFrequency can’t affect the framerate, it is normally a constant.
GevSCPD can, but only if you are using several camera on the same network.

Is the camera directly connected to your machine ?

Try to debug your issue without using container or virtualisation, they add another layer of complexity that does not help.

Did you try to enable the debug output ? ARV_DEBUG=device should give you some statistics about packet transfers at the end of the run. Something like:

 26 frames/s -    6.82 MiB/s
 25 frames/s -    6.55 MiB/s
 25 frames/s -    6.55 MiB/s
^C 25 frames/s -    6.55 MiB/s
n_completed_buffers    = 101
n_failures             = 0
n_underruns            = 0
n_timeouts             = 0
n_aborteds             = 0
n_missing_frames       = 0
n_size_mismatch_errors = 0
n_received_packets     = 19695
n_missing_packets      = 0
n_error_packets        = 0
n_ignored_packets      = 0
n_resend_requests      = 0
n_resent_packets       = 0
n_resend_ratio_reached = 0
n_resend_disabled      = 0
n_duplicated_packets   = 0
n_transferred_bytes    = 26638144
n_ignored_bytes        = 0
10

Does the problem disappear if you run the pipeline to fakesink rather than appsink ?

11

might be called GainRaw on basler camera in the genicam tree.
might also work with gain=(integer_value > 0) in the pipeline template.

13

We haven’t tried this before. Will see if we can try that later.

14

From camera to the host, there were no packets dropped, and there were always some packets dropped from the host to the container, even the frame rate was 1.0.

The network packets we saw:

frame rate, packets captured, packets received, packets dropped
0.7, 295972 , 309309 , 13334 , 4.3% dropped
0.7 , 298172 , 325061 , 26883 , 8.2% drooped
0.75 , 371772 , 405773 , 33956 , 8.3% drooped
1.0 , 468096 , 482954 , 14842 , 3% dropped

Since the issue happened every few hours, i was thinking there was probably some system update or antivius running, but i didn’t see any.

BTW, if the camera is using interface 0, and we saw extra network traffic on interface 1, will that affect the frame rate?

15

When i ran arv-test-0.6, i got following output which was the same as another camera which had good frame rate:

[GvDevice::new] Interface address = 192.168.0.254
[GvDevice::new] Device address = 192.168.1.67
[GvDevice::load_genicam] xml url = ‘Local:Basler_Ace_GigE_66658f4d_Version_3_6.zip;c0000000;11710’ at 0x200
[GvDevice::load_genicam] Xml address = 0xc0000000 - size = 0x11710 - Basler_Ace_GigE_66658f4d_Version_3_6.zip
[GvDevice::load_genicam] Zipped xml data
[GvDevice::write_register] Unexpected answer (0x8006)
[GvDevice::write_register] Unexpected answer (0x8006)
[GvDevice::write_register] Unexpected answer (0x8006)
[GvDevice::write_register] Unexpected answer (0x8006)
[GvDevice::write_register] Unexpected answer (0x8006)
[GvDevice::take_control] Can’t get control access
[GvDevice::new] Packet resend = yes
[GvDevice::new] Write memory = yes
[GvDevice::new] Register workaround = yes

(process:82682): Aravis-CRITICAL **: 17:07:33.019: arv_xml_schema_validate: assertion ‘schema->priv->valid_ctxt != NULL’ failed

(process:82682): Aravis-CRITICAL **: 17:07:33.019: arv_xml_schema_validate: assertion ‘schema->priv->valid_ctxt != NULL’ failed
Genicam: FAILED Invalid Genicam XML data
Sensor size read: 1282 1026
[GvDevice::write_register] Unexpected answer (0x8006)
[GvDevice::write_register] Unexpected answer (0x8006)
[GvDevice::write_register] Unexpected answer (0x8006)
[GvDevice::write_register] Unexpected answer (0x8006)
[GvDevice::write_register] Unexpected answer (0x8006)

16

I do not think the framrate is going down due to increased exposure time,
rather, either the processing in appsink clogs the system up, or you are having some network issues.
Your payload size is quite small so you shoud not be maxing out the connection.
Check with fakesink. Should help diagnose.

17

I did try fakesink, but got error: Failed to open the frame source for camera
which means our VideoCapture couldn’t open.

18

Well, fakesink is just a black hole, capturing from it would be tricky.
The point of using it it to see if the problem lies with camera or the processing of frames.
if the frame processing requirement by appsink varies, the pipe could get clogged.

if you replace your source with a testsource (not sure there is a bayer one, so you might need to adjust your pipeline accordingly.

Another way to track down the problem could be to capture to disk, then replay it into your pipeline.

Other options include to time your appsink loop. (does it take to long?)
Sometimes it helps with inserting a tee and queue elements in the pipeline (multiple threads) for speedup.