Certain features of RKHM enable full automation of content generation. In this article, we will guide you on how to leverage RKHM using third-party APIs and software to create data effectively.

In the context of music, content automation can be broadly divided into two main categories:

1. Collecting data ( artist-name, artist-cover, album-title, track-list, track-title and etc )
   
2. Collecting source ( something with sound! An audio, video, YouTube or Soundcloud frame )
   

Data collection

For first part, which is data collection, the main source is "Spotify". While other sources, like Musixmatch can be used for lyrics and you can use AI ( through LoremAI plugin ) to generate addiotional data ( description, cover, etc ) for items gathered from Spotify

You can approach data collection from Spotify in 2 ways. Each one has their own pros & cons

Full automation

Full automation allows RKHM to automatically update and sync music data with Spotify in real-time. This means whenever a user searches for music or visits an artist's page on your website, RKHM will immediately request and retrieve the latest data from Spotify.

How it Works:

• When a user searches on your site, RKHM sends this query to Spotify's API to get the search results.
• These results are then displayed on your site. If users click on an artist, album, or track that doesn't already exist on your site, RKHM will create it and show it to the user instantly.
• If a user visits an existing artist's page, RKHM ensures that all the latest releases from that artist are included by checking with Spotify.

This option works great, unless:

1. High Traffic Considerations: If your site attracts thousands of visitors each day, it's possible to hit Spotify's API limits.
2. Prioritizing Speed: To ensure RKHM operates swiftly, it’s beneficial to pre-collect data from Spotify with cronjobs. Fetching data on-demand can introduce delays, as each user request requires time to communicate with Spotify's servers.
   
3. Targeted Content Selection: If full automation is active, any search by users can bring all corresponding Spotify content to your site, which might include undesired or irrelevant items. To maintain a curated selection of content, especially if targeting a specific niche, consider employing semi-automation or setting up cronjobs.
   
4. Managing Source Collection: With full automation, your site might automatically create entries for new data not previously existent in your database, as triggered by user searches. These entries often need associated media sources, which requires additional time if your script needs to fetch these from YouTube or SoundCloud. For a more efficient process, particularly if you prefer to download videos directly rather than using embedded YouTube frames, setting up cronjobs allows you to have all necessary resources ready and waiting for when users visit your site.
   

Semi Automation

The second option available is "Semi automation" or what can be termed as "Music Cronjob". This method utilizes cronjobs to automatically gather data from Spotify in the background. Whether you aim to gather a massive amount of data (up to over 100,000,000 tracks on Spotify) or focus on a more specific niche, this method can be tailored to your needs. You can choose to sync selective content such as certain artists, playlists, or genres.

Think of music cronjobs as diligent carriers working behind the scenes. These cronjobs travel to Spotify, collect the necessary data, and transport it back to your server without manual intervention. You can control what they collect, how often they do it, and the volume of data they should fetch. Naturally, the more data you desire, the more cronjobs you'll need to deploy, which translates into more API requests and database operations.

With cronjobs, you have the option to operate with "Static IDs" that you specify or have RKHM dynamically fetch these IDs from your database based on defined criteria. However, it's important to manage the number of active cronjobs judiciously to avoid overwhelming the system, especially when using "Dynamic IDs". For instance, if you have 100 artists in your database, you could set a cronjob to synchronize all these artists with Spotify weekly, fetching their albums and related artists. This dynamic fetching can result in the creation of new artist entries as the system discovers affiliated artists, potentially causing the cronjob to continually expand its scope of operation.

Moreover, cronjobs don't merely collect the data—they help maintain its freshness. For example, they can continuously check if a popular artist like Eminem has dropped a new track. However, it's worth noting that most elements, including the vast majority of albums, tracks, and artists, do not need frequent updates. Typically, a single collection cycle, followed by periodic checks for updates in popularity or new releases, is sufficient.

When setting up your music cronjobs, there are two critical factors to consider:

1. Spotify API Limit: With a standard free API key, you can make up to 180 requests per minute to Spotify. This capacity is generally adequate for most projects unless you aim to process the entirety of Spotify's catalog. If needed, you can utilize tools like "Spotify Multi Account" to employ multiple API keys with proxies to effectively navigate around Spotify’s API constraints.
2. Server Limits: Although it's technically feasible to synchronize data round-the-clock, it's essential to balance this ambition with the actual capabilities and limits of your server to ensure smooth operation without resource exhaustion.
   

By understanding and configuring these elements carefully, you can optimize the performance and efficiency of your music cronjobs, ensuring they serve your project’s needs without causing system strain.

To optimize API usage and conserve server resources, it's crucial to carefully plan your cronjobs. When setting up a new cronjob, you have several configuration options that can significantly impact efficiency, including "Execution Interval," "Update Interval," and "Item Limit."

Execution Interval: This setting determines how frequently the cronjob runs, measured in minutes. For instance, setting it at 60 means the cronjob will run up to 24 times per day. You might adjust this number based on other cronjobs running concurrently since they are executed sequentially. For less frequent needs, a higher interval number reduces the frequency of execution, conserving API requests. If unsure, a starting interval of 10 minutes is a good baseline.

Update Interval: This setting controls how current the data stays by defining how often the data should sync with Spotify. A lower number in this setting means more frequent updates—for example, setting it at 1 day will resync the data daily, while setting it at 365 will refresh it annually. Frequent updates should only be set for dynamic content such as popular artists with frequent new releases. For most tracks and albums, whose popularity doesn't change drastically, a higher update interval is more suitable and also reduces the burden on your API quota.

Item Limit: This option specifies the number of items processed per execution. A lower number minimizes the number of API requests per job run. If unsure, setting the item limit to 1 can help manage API use. Note that one item does not necessarily equate to one API request; some items, like user playlists, may require multiple requests per item.

Additionally, consider the nature of the data when configuring these settings. For example:

User Playlists: Syncing a single playlist could net hundreds or even thousands of API requests if the playlist contains numerous items.

General Advice: We would suggest that you create multiple cronjobs for different purposes. We suggest that you set "item limit" for all cronjobs to "1", set the "execution interval" to 10 and enter different "update interval" for different cronjobs based on how often they might get refreshed on Spotify

By implementing these strategies, you can streamline your resource usage while ensuring your data remains up-to-date and relevant. Tailoring each cronjob to its specific purpose and the frequency of data changes on Spotify will lead to more efficient operations.

Semi Automation: Examples

Example 1: Syncing Hip-Hop Artists Begin by creating a cronjob named "Seed" which functions to fetch data daily from Spotify for your favorite hip-hop artists. This job will require setting the update interval to 1 for daily updates and configuring the API settings to Spotify with the object type set to artist. You should enable features like "Get related artists," "Get albums," and "Sync genres." Utilize the "browse Spotify" button in the "query" tab to select and add the IDs of 10 favored artists. This will allow the cronjob to collect all relevant data including albums, tracks, and up to 200 related artists for each selected artist.

For weekly updates on highly popular artists in your database, set up another cronjob named "Famous Checker." This should update on a weekly basis (update interval of 7) and use dynamic IDs to select artists with a Spotify popularity minimum of 70. This ensures that your database remains synced with all data from highly popular artists.

Additionally, create a "Mid Checker" cronjob for bi-monthly updates on moderately popular artists. Set the update interval to 60 days and adjust the criteria to select artists with a Spotify popularity minimum of 40. This setup helps to keep your database fresh with current information on moderately popular artists.

Example 2: Syncing Famous Albums For weekly collection of data from Spotify’s key playlists, you can establish a cronjob named "Seed" configured for playlists instead of artists. With an update interval of 7 (weekly), set the API Name to Spotify and the Object Type to User Playlists, and remember to enable "sync genres." By entering "Spotify" as a static ID in the "query" tab, this job will collect all official and notable playlists which feature famous songs.

To ensure that albums are comprehensive in your database, set up an "Album Getter" cronjob designed to sync albums included in playlists to fetch any missing tracks. This job only needs to be updated once a year (update interval of 365), given that albums generally do not require frequent updates. Adjust the settings to fetch albums and enable dynamic IDs to keep the album content fresh and inclusive of all possible tracks.

These methods show how semi-automation can efficiently manage music data in RKHM using Spotify. Using structured cronjobs, you can maintain a relevant and up-to-date music database, tailoring the data collection to specific needs while ensuring efficient use of your server and API resources. Each example demonstrates that while the strategies are diverse, the underlying principle remains to wisely manage API calls and maintain data relevance to your platform's focus.

Source collection

The second critical component in fully automating your music platform with RKHM involves collecting media sources for the items you've gathered or generated data for. RKHM supports multiple methods to automate the collection of sources, ensuring that your content is not only abundant in information but also rich in media.

1. YouTube Video: RKHM allows you to set up automated tasks to find related YouTube videos for each item either on user demand or automatically in the background through cronjobs. This can be particularly useful for enhancing user engagement by providing video content directly related to user searches or featured lists on your platform.

2. SoundCloud Audio: For platforms focusing on audio tracks, RKHM offers the capability to automatically fetch corresponding SoundCloud audio files for any listed item per user request. Currently, this function does not support background automation via cronjobs, but enhancements for such capabilities are under consideration.

3. YouTube Video to MP3 ( youtube-dl ) : If your platform benefits from offering downloadable audio files, RKHM can be configured to source YouTube videos and convert them into MP3 format. This is feasible both as a direct response to user actions and via cronjobs set to run these tasks in the background. Given the processing time required for downloading and converting videos, setting up cronjobs for batch processing is advisable to avoid latency during user interaction. Ensure the use of "filters" when configuring these cronjobs to prevent the system from downloading and converting every video associated with tracks in your database, thus managing your resources effectively.

4. Audio Files from Torrents: Expanding your library with diverse sources, RKHM supports integration with the Music Torrent Plugin. This plugin allows RKHM to run in the background and fetch audio files from popular torrent indexing websites using the Transmission BitTorrent client. This method is incredibly useful for acquiring rare tracks or high-quality audio files that are not readily available through conventional streaming services.

Each of these methods provides a robust framework for source collection, allowing your platform not only to offer extensive metadata on music tracks, artists, and albums but also to enrich the user experience with multimedia content. It is crucial to configure these settings thoughtfully, considering both the nature of your platform and the expectations of your users, to optimize performance and enhance content delivery dynamically.

When it comes to source automation, you have two options just like the data collection. One being handling user requests live, the other one being pre-collecting sources using cronjobs. For Youtube & Soundcloud, both options are fine, since once the ID of item is found, it is stored in your database and no more API requests are made for that item. For youtube-dl & torrent, we highly suggest using cronjobs rather than service requests live as youtube-dl requires time to fulfil requests

Make sure you enter a big "update interval" for source-collecting cronjobs since you don't need to re-collect an already-stored source!

Dealing with limits

You can check `Cronjob logs` or `Error_logs` in admin-area to see if you have reached any limits

Spotify: For data collection, Spotify's API is used. Spotify has very relaxed limits, so you should not reach the limit unless you are syncing too many items or too frequently. You can reach out to Spotify to increase your limit legally, or use "Spotify multi account" tool to define multiple API keys ( with proxies ) to increase your limit illegally

YouTube: Compared to Spotify, YouTube has much limited API. You can request higher limits from Google, pass multiple API keys or use `YouTube scrapper` to bypass limits altogether

SoundCloud: SoundCloud does not give API keys anymore but you can install "Soundcloud scrapper" to bypass limits altogether