How to manage two cameras connected to different network interfaces, with link-local IP

zjeffer · September 11, 2024, 9:17am

I have a PCIe ethernet card with two gigabit ports.
Each port has a GigECamera connected in link-local mode.

I gave each interface its own 169.254.x.x/16 IP address:

4: enp138s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8192 qdisc mq qlen 1000
    link/ether 74:fe:48:68:a3:65 brd ff:ff:ff:ff:ff:ff
    inet 169.254.0.1/16 brd 169.254.255.255 scope global enp138s0f0
       valid_lft forever preferred_lft forever
    inet6 fe80::76fe:48ff:fe68:a365/64 scope link 
       valid_lft forever preferred_lft forever
5: enp138s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8192 qdisc mq qlen 1000
    link/ether 74:fe:48:68:a3:66 brd ff:ff:ff:ff:ff:ff
    inet 169.254.1.1/16 brd 169.254.255.255 scope global enp138s0f1
       valid_lft forever preferred_lft forever
    inet6 fe80::76fe:48ff:fe68:a366/64 scope link 
       valid_lft forever preferred_lft forever

The cameras are both detected and get a link-local IP address:

# arv-tool-0.8 
Basler-a2A2590-22gmBAS-40217775 (169.254.219.121)
NET GmbH-GP4136C-154124002001 (169.254.139.118)

I cannot connect to one of the cameras (the Basler one). In ip route, I see:

# ip route
default via 192.168.254.241 dev eno2  metric 200 onlink 
169.254.0.0/16 dev enp138s0f0 scope link  src 169.254.0.1 
169.254.0.0/16 dev enp138s0f1 scope link  src 169.254.1.1 
192.168.254.240/28 dev eno2 scope link  src 192.168.254.251

Meaning enp138s0f0 (where the NET camera is connected to) is always the first interface to respond. While the Basler camera is connected to enp138s0f1, I think aravis will try to connect to it through the first interface it finds (because it’s the same subnet).

If I execute ip link set enp138s0f0 down, the Basler camera is usable. If I then set enp138s0f1 up again, the Basler camera still works but now the NET doesn’t, and the ip route order has changed:

# ip route
default via 192.168.254.241 dev eno2  metric 200 onlink 
169.254.0.0/16 dev enp138s0f1 scope link  src 169.254.1.1 
169.254.0.0/16 dev enp138s0f0 scope link  src 169.254.0.1 
192.168.254.240/28 dev eno2 scope link  src 192.168.254.251

Is there a way to tell aravis to connect through a specific network interface? Or is there another way to handle this?

eudoxos · September 17, 2024, 9:56am

Have a look here, that’s really addressing issue. linux - Link-local (zeroconf) networking with multiple hosts/interfaces - Super User . Options are:

create a bridge interface (the linked article talks about that);
get a cheap switch, connect only one of the link-local interfaces to the switch, and both cameras. Plus set the other iface to something else than link-local. This is what I would do.

zjeffer · September 23, 2024, 3:01pm

Hi @eudoxos, thanks for your suggestions.

The cheap switch option is not really possible because it’s an embedded system and we can’t add new hardware.

I managed to do it with the bridge interface, I’ll update this post soon with detailed instructions once I have some more time.

Thank you very much!

zjeffer · September 24, 2024, 4:07pm

Here’s what I’m doing:

ip link add name br0 type bridge
ip link set enp138s0f0 master br0
ip link set enp138s0f1 master br0

and this is in my /etc/network/interfaces:

# enp138s0f[0-1] are the PoE interfaces
allow-hotplug enp138s0f0
iface enp138s0f0 inet static
  address 169.254.100.1
  netmask 255.255.0.0
  mtu 8192
  metric 400

allow-hotplug enp138s0f1
iface enp138s0f1 inet static
  address 169.254.200.1
  netmask 255.255.0.0
  mtu 8192
  metric 400

# bridge over PoE interfaces to allow communication with the cameras
auto br0
iface br0 inet static
  address 169.254.1.0
  netmask 255.255.0.0
  bridge_ports enp138s0f0 enp138s0f1
  mtu 8192
  metric 300

# ip route
default via 192.168.254.241 dev eno2  metric 200 onlink
169.254.0.0/16 dev br0 scope link  src 169.254.1.0
169.254.0.0/16 dev enp138s0f0 scope link  src 169.254.100.1
169.254.0.0/16 dev enp138s0f1 scope link  src 169.254.200.1
192.168.254.240/28 dev eno2 scope link  src 192.168.254.251

Now the cameras can both be connected to at the same time, and using the arv_gv_device_get_interface_address function I can ask which interface they belong to. When executing the function on the camera connected to enp138s0f0, it will return 169.254.100.1, the other one will return 169.254.200.1. Great: I can use this to differentiate from software which camera is which.

There’s a problem with capturing, though: I notice my own application outputs this at startup:

[15:51:12.969] _ interface> Found 0 USB3Vision device (among 3 USB devices)                                        
[15:51:12.970] _ interface> [GvDiscoverSocket::new] Add interface 127.0.0.1 (127.0.0.1)                            
[15:51:12.970] _ interface> [GvDiscoverSocket::new] Add interface 192.168.254.251 (192.168.254.255)                
[15:51:12.970] _ interface> [GvDiscoverSocket::new] Add interface 169.254.100.1 (169.254.255.255)                  
[15:51:12.970] _ interface> [GvDiscoverSocket::new] Add interface 169.254.200.1 (169.254.255.255)                  
[15:51:12.970] _ interface> [GvDiscoverSocket::new] Add interface 169.254.1.0 (169.254.255.255)                    
[15:51:12.971] _ interface> [GvInterface::discovery] Device 'Basler-a2A2590-22gmBAS-40217775' found (interface 169.254.1.0) user_id 'Basler-2' - MAC '00:30:53:37:02:a1'                                                              
[15:51:12.971] _ interface> [GvInterface::discovery] Device 'Basler-a2A2590-22gmBAS-40217775' found (interface 169.254.200.1) user_id 'Basler-2' - MAC '00:30:53:37:02:a1'                                                            
[15:51:13.972] _ interface> Found 0 USB3Vision device (among 3 USB devices)                                        
[15:51:13.972] _ device> [GvDevice::new] Interface address = 169.254.200.1                                         
[15:51:13.972] _ device> [GvDevice::new] Device address = 169.254.138.52                                           
[15:51:13.974] _ device> [GvDevice::load_genicam] xml url = 'Local:genicam_789bc3c6.zip;c0000000;ce36' at 0x200

For the Basler camera, it finds two interfaces:

169.254.1.0 (br0)
169.254.200.1 (enp138s0f1)

[15:51:13.972] _ device> [GvDevice::new] Interface address = 169.254.200.1 shows me the correct interface address of the enp138s0f1 interface.

Captures do not work at all with this: the buffer it returns is always null.

[16:07:02.584] _ stream> [GvStream::finalize] n_transferred_bytes    = 0
[16:07:02.584] _ stream> [GvStream::finalize] n_ignored_bytes        = 0
[16:07:02.584] _ stream> [Stream::finalize] Flush 1 buffer[s] in input queue
[16:07:02.585] _ stream> [Stream::finalize] Flush 0 buffer[s] in output queue
Error acquiring image: buffer is null

When I try it with arv-camera-test-0.8, this is the output at startup:

[16:04:47.232] _ interface> Found 0 USB3Vision device (among 3 USB devices)                                           
[16:04:47.232] _ interface> [GvDiscoverSocket::new] Add interface 127.0.0.1 (127.0.0.1)                               
[16:04:47.232] _ interface> [GvDiscoverSocket::new] Add interface 192.168.254.251 (192.168.254.255)                   
[16:04:47.232] _ interface> [GvDiscoverSocket::new] Add interface 169.254.100.1 (169.254.255.255)                     
[16:04:47.233] _ interface> [GvDiscoverSocket::new] Add interface 169.254.200.1 (169.254.255.255)
[16:04:47.233] _ interface> [GvDiscoverSocket::new] Add interface 169.254.1.0 (169.254.255.255)
[16:04:47.233] _ interface> [GvInterface::discovery] Device 'Basler-a2A2590-22gmBAS-40217775' found (interface 169.254.1.0) user_id 'Basler-2' - MAC '00:30:53:37:02:a1'
[16:04:47.233] _ device> [GvDevice::new] Interface address = 169.254.1.0
[16:04:47.234] _ device> [GvDevice::new] Device address = 169.254.138.52

The Basler device is only found once
The interface address is now 169.254.1.0 (br0), so I can’t use this to find which camera this is.

Captures work fine:

[16:04:54.997] _ stream> [GvStream::finalize] n_transferred_bytes    = 354139188
[16:04:54.997] _ stream> [GvStream::finalize] n_ignored_bytes        = 0

What does arv-camera-test do that causes it to only find one interface (br0), but causes the captures to work fine?

zjeffer · October 4, 2024, 2:59pm

I removed the IP addresses from the enp138s0f0 and f1 interfaces, so both camera’s get 169.254.1.0 as the interface address. Communication works but I have to hardcode the camera identification using the camera brand (one of the camera’s is always the same brand). This will do for now. I’ll look for a solution to this at a later time.

I’m having some trouble handling disconnects & reconnects with this bridge, though.
After disconnecting the camera, I get the control-lost signal but the camera keeps returning Ack reception timeouts:

[14:52:13.946] _ device> [GvDevice::read_memory] Ack reception timeout                                                                                                                                                                                       
[14:52:13.946] _ device> [GvDevice::read_memory] Ack reception timeout                                                                                                                                                                                       
[14:52:14.447] _ device> [GvDevice::read_memory] Ack reception timeout                                                                                                                                                                                       
[14:52:14.447] _ device> [GvDevice::read_memory] Ack reception timeout                                                                                                                                                                                       
[14:52:14.948] _ device> [GvDevice::read_memory] Ack reception timeout                                                                                                                                                                                       
[14:52:14.948] _ device> [GvDevice::read_memory] Ack reception timeout                                                                                                                                                                                       
[14:52:15.448] _ device> [GvDevice::read_memory] Ack reception timeout                                                                                                                                                                                       
[14:52:15.448] _ device> [GvDevice::read_memory] Ack reception timeout                                                                                                                                                                                       
[14:52:15.949] _ device> [GvDevice::read_memory] Ack reception timeout                                                                                                                                                                                       
[14:52:15.949] _ device> [GvDevice::read_memory] Ack reception timeout                                                                                                                                                                                       
14:52:15,949  Gain: GigEVision read_memory timeout                                                                                                          
[14:52:16.450] _ device> [GvDevice::read_memory] Ack reception timeout                                                                                                                                                                                       
[14:52:16.450] _ device> [GvDevice::read_memory] Ack reception timeout                                                                                                                                                                                       
[14:52:16.950] _ device> [GvDevice::read_memory] Ack reception timeout                                                                                                                                                                                       
[14:52:16.950] _ device> [GvDevice::read_memory] Ack reception timeout                                                                                                                                                                                       
[14:52:17.451] _ device> [GvDevice::read_memory] Ack reception timeout                                                                                                                                                                                       
[14:52:17.451] _ device> [GvDevice::read_memory] Ack reception timeout                                                                                                                                                                                       
[14:52:17.952] _ device> [GvDevice::read_memory] Ack reception timeout                                                                                                                                                                                       
[14:52:17.952] _ device> [GvDevice::read_memory] Ack reception timeout                                                                                                                                                                                       
[14:52:18.453] _ device> [GvDevice::read_memory] Ack reception timeout                                                                                                                                                                                       
[14:52:18.453] _ device> [GvDevice::read_memory] Ack reception timeout                                                                                                                                                                                       
14:52:18,453  Gamma: GigEVision read_memory timeout

It seems to do this ad infinitum. I’m not sure what part of my application is trying to continuously read the camera’s properties (gain, gamma, exposure, etc). Will investigate this.
When it’s in this state (after reconnecting the camera) taking control access with arv_gv_device_take_control doesn’t work:

[14:55:26.898] _ device> [GvDevice::take_control] Can't get control access`

I assume I’ll have to hook up the control-lost signal, and recreate the ArvCamera using arv_camera_new?

zjeffer · November 4, 2024, 8:44am

I assume I’ll have to hook up the control-lost signal, and recreate the ArvCamera using arv_camera_new?

This was indeed the fix. It’s quite slow to reconnect but adding the control-lost signal and destroying the ArvCamera object when it lost control works.

I still can’t differentiate between the different cameras, but for now I can hardcode it based on the camera brand.