Fitness landscapes (FL) datasets from mutational studies are now commonplace and allow us to investigate the subtle interplay of genotype-phenotype mapping. These approaches form a very powerful tool in system biology, where these genetic perturbations reveal the underlying molecular pathways controlling the phenotype. Modeling such FL has the potential to inform us of the pathways hidden behind fitness perturbations. Two somewhat opposed strategies for FL modeling can be found in the literature: i) We can use hypotheses about the pathways to create a biophysical model. Such an approach possesses very good interpretability and fitting properties. However, it requires access to prior information on pathways. ii) Machine learning (ML) offers a powerful agnostic top-down approach where mutational effects are well reproduced. However, such methods are usually not interpretable and have poor extrapolation power---predicting novel genotype perturbations. In this work, we investigated an ML-based framework for modeling gene perturbation fitness landscapes. Our framework displayed comparable performances in fitness prediction with state-of-the-art biophysical models on a two-gene system. Besides, we showed that informing the framework with biophysical hypotheses improved its performance when only a few measurements of fitness were available. This low data regime is critical as most datasets may only characterize a very small fraction of possible perturbations. Based on the observation that some perturbations have similar effects, we devised a continuity constraint to improve the model's performance in low-data regime situations. Our ongoing efforts aim at using this framework to categorize the modeled fitness landscapes based on their complexity. Indeed, fitness landscapes may display various epistatic patterns that unveil complex gene interactions.