Context:

Kolmogorov-Arnold Networks recently receive increased attention [1], as a promising alternative to the traditional Multi-Layer Perceptrons (MLPs) that currently power many deep learning applications.

In MLPs, nodes (neurons) perform pre-defined, non-linear activation functions on the weighted sum of their input edges (connections). The linear weights on the connections are adjusted during training.

Instead, KAN models have learnable activation functions on edges, which are basis functions for example B-splines. By learning the coefficients of these functions, KANs can achieve highly flexible and expressive activation functions on each connection. Then KAN neurons simply sum up the values on their input edges.

 The KAN model have potential to offer better performance at a smaller model size. The parametrized basis functions are much more expressive than a series of learned linear projections followed by fixed, non-linear activations. Therefore, less of such expressive functions may be sufficient to construct a model to solve a given problem. Moreover, as the number of basis functions are smaller, the KAN solutions can be more explicable than large MLPs.

 Nevertheless, KANs have also downsides. As the basis functions are more complex, the literature reports longer training times; furthermore good hyperparameters for the models are more difficult to identify. Another important issue is the risk of overfitting. As the basis functions are more complex, they can easily overfit any data for the trainining dataset.

Intership objectives:

In this context, the main question addressed in this internship is: what is the potential of Kolmogorov-Arnold Networks in embedded Artificial Intelligence (AI) systems? Are they really more performant and more efficient (in terms of compute resources necessary for inference)?

 The investigation will be carried out on time-series tasks, such as prediction or detection. A typical candidate application is keyword spotting. Two versions of this application could be studied, evaluated and compared: a KAN [2] and a transformer-based implementation [3].

References:

[1] Liu, Ziming, et al. "Kan: Kolmogorov-arnold networks," arXiv preprint arXiv:2404.19756 (2024)

[2] Xu, Anfeng, et al. "Effective Integration of KAN for Keyword Spotting," arXiv preprint arXiv:2409.08605 (2024)

[3] A. Berg, M. O’Connor, et M. T. Cruz, “Keyword Transformer: A Self-Attention Model for Keyword Spotting,” arXiv preprint arXiv:2104.00769 (2021)

Expected skills:

• Last year of Engineering school or Master
• Skills in embedded systems, programming, scientific computing
• Knowledge on artificial intelligence, neural networks
• Python/C/C++ programming
• Experience on PyTorch

Plan of work:

• Familiarize with KAN: state-of-the-art publications and tools and software libraries (based on Pytorch)
• Define metrics for hardware implementation cost
• Propose a method to determine the computation cost of existing networks- Profile KAN and conventional DNNs for a specific problem based on existing profiling tools
• AN trade-offs: hardware cost vs. performance (accuracy, detection error rate,…)
• Write the final report, prepare the presentation