- Select a similar modeling problem 
- Create a complex model for the first task
- Reuse model from the first task on second task
- Tune and perfect the model



1) Developing your own model


https://machinelearningmastery.com/transfer-learning-for-deep-learning/

Approaches to Transfer Learning

Another approach is to use deep learning to discover the best representation of your problem, which means finding the most important features. This approach is also known as representation learning, and can often result in a much better performance than can be obtained with hand-designed representation.
In machine learning, features are usually manually hand-crafted by researchers and domain experts. Fortunately, deep learning can extract features automatically. Of course, this doesn't mean feature engineering and domain knowledge isn’t important anymore — you still have to decide which features you put into your network. That said, neural networks have the ability to learn which features are really important and which ones aren’t. A representation learning algorithm can discover a good combination of features within a very short timeframe, even for complex tasks which would otherwise require a lot of human effort.


FEATURE EXTRACTION

Pre-training in deep learning involves optimizing a neural network on an initial task before it is fine-tuned for a specific application. The core idea is that a model trained for one purpose can be successfully adapted for another. This strategy avoids the need to build complex models from the ground up, a significant advantage when dealing with tasks that have scarce labeled data. By utilizing tasks with more abundant supervision signals, pre-training reduces dependence on task-specific labels and facilitates the creation of more versatile, general-purpose models.

Pre-training in Deep Learning

- Developing your own model
- Using a pre-trained model

University of Michigan - Ann Arbor

The longest part of the development of a deep learning model is that of the training stage. However, if the problem that is being tackled has already been studied and worked on by someone else and the model has been released to the public including the trained weights then a technique to substantially increase the training process is to utilize transfer learning.

Transfer Learning in Deep Learning

https://towardsdatascience.com/transfer-learning-using-the-fastai-library-d686b238213e

Transfer Learning Reference

- Need very large dataset
- Data must be general and not domain-specific
- Only works if initial and current problem are similar enough


Disadvantages of Transfer Learning

According to DeepMind CEO Demis Hassabis, transfer learning is also one of the most promising techniques that could lead to artificial general intelligence (AGI) someday.

ideas about transfer learning from the specialist

The primary advantages of transfer learning in deep learning include reduced training time, improved neural network performance, and lower data requirements. Since deep neural networks typically require massive datasets and extensive computational time to train from scratch, leveraging a pre-trained model allows for the development of robust models even when domain-specific data is scarce or expensive to acquire.

Advantages of Transfer Learning in Deep Learning

A practical example of fine-tuning involves adapting an image classification model. First, a source model is pre-trained on a large source dataset, such as ImageNet. To adapt this model for a specific task, like classifying images in a smaller target dataset of chairs, a target model is constructed by copying the source model's parameters and replacing its output layer. The new output layer is then trained from scratch on the chair dataset, while the remaining layers are fine-tuned, allowing the target model to leverage the broad visual features learned from the source dataset.

Example of Fine-Tuning for Image Classification

To classify a subset of a large dataset—such as a specific group of classes from ImageNet—it is computationally advantageous to use models pre-trained on the full dataset to extract features. Rather than training a model from scratch or fine-tuning all layers, the pre-trained architecture can be utilized, and only a custom, small-scale output network needs to be trained on the subset. This approach leads to less computational time and memory cost while capitalizing on the robust visual representations already learned by the pre-trained model.

Efficient Subset Classification via Pretrained Feature Extraction

Transfer learning from a source task (Task A) to a target task (Task B) in deep learning is most effective under the following conditions:
1. Tasks A and B share the same input $$x$$.
2. There is significantly more training data available for Task A than for Task B.
3. Low-level features learned from Task A are expected to be helpful for learning Task B.

Learn Before

Related

Learn After