Hello,
The AM335x TRM states in "26.1 Functional Description"
The device always starts up in secure mode. The ROM Code takes care of early initialization. The ROM code switches the device into public mode. Hence the Public ROM Code provides run-time services for cache maintenance.
Sounds to me like the operation mode on the PandaBoard, so, the kernel should operate as if TrustZone is not there except for cache handling SMCs.
Regards