Torch is a framework for scientific computations with wide support for machine learning written in Lua. It is one of the most popular deep learning frameworks. Its supported platforms are Linux, macOS, Windows, Android, and iOS, and it also supports acceleration with CUDA and OpenCL (partially). There are numerous third-party packages that introduce additional capabilities to Torch. Autograd introduces automatic differentiation, nn package allows construct neural networks from simple building blocks, the rnn package provides recurrent neural networks, and iTorch provides interoperability with IPython Notebook. The Caffe model can be loaded using the loadcaffe package. Its library is well documented and easy to install.
The main problem for Swift developers will be the Lua language itself, because its paradigm is quite different from Swift's one; however, there are libraries that introduce strong typing and functional capabilities to Lua to make life less painful.