Skip to content

Commit

Permalink
initial commit
Browse files Browse the repository at this point in the history
  • Loading branch information
@ilaria-manco
ilaria-manco committed Jul 19, 2024
0 parents commit cbdf41a
Show file tree
Hide file tree
Showing 29 changed files with 3,327 additions and 0 deletions.
1 change: 1 addition & 0 deletions .nojekyll
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@

48 changes: 48 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
# Academic Project Page Template
This is an academic paper project page template.


Example project pages built using this template are:
- https://vision.huji.ac.il/spectral_detuning/
- https://vision.huji.ac.il/podd/
- https://dreamix-video-editing.github.io
- https://vision.huji.ac.il/conffusion/
- https://vision.huji.ac.il/3d_ads/
- https://vision.huji.ac.il/ssrl_ad/
- https://vision.huji.ac.il/deepsim/



## Start using the template
To start using the template click on `Use this Template`.

The template uses html for controlling the content and css for controlling the style.
To edit the websites contents edit the `index.html` file. It contains different HTML "building blocks", use whichever ones you need and comment out the rest.

**IMPORTANT!** Make sure to replace the `favicon.ico` under `static/images/` with one of your own, otherwise your favicon is going to be a dreambooth image of me.

## Components
- Teaser video
- Images Carousel
- Youtube embedding
- Video Carousel
- PDF Poster
- Bibtex citation

## Tips:
- The `index.html` file contains comments instructing you what to replace, you should follow these comments.
- The `meta` tags in the `index.html` file are used to provide metadata about your paper
(e.g. helping search engine index the website, showing a preview image when sharing the website, etc.)
- The resolution of images and videos can usually be around 1920-2048, there rarely a need for better resolution that take longer to load.
- All the images and videos you use should be compressed to allow for fast loading of the website (and thus better indexing by search engines). For images, you can use [TinyPNG](https://tinypng.com), for videos you can need to find the tradeoff between size and quality.
- When using large video files (larger than 10MB), it's better to use youtube for hosting the video as serving the video from the website can take time.
- Using a tracker can help you analyze the traffic and see where users came from. [statcounter](https://statcounter.com) is a free, easy to use tracker that takes under 5 minutes to set up.
- This project page can also be made into a github pages website.
- Replace the favicon to one of your choosing (the default one is of the Hebrew University).
- Suggestions, improvements and comments are welcome, simply open an issue or contact me. You can find my contact information at [https://pages.cs.huji.ac.il/eliahu-horwitz/](https://pages.cs.huji.ac.il/eliahu-horwitz/)

## Acknowledgments
Parts of this project page were adopted from the [Nerfies](https://nerfies.github.io/) page.

## Website License
<a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/"><img alt="Creative Commons License" style="border-width:0" src="https://i.creativecommons.org/l/by-sa/4.0/88x31.png" /></a><br />This work is licensed under a <a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/">Creative Commons Attribution-ShareAlike 4.0 International License</a>.
272 changes: 272 additions & 0 deletions index.html
Original file line number Diff line number Diff line change
@@ -0,0 +1,272 @@
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<!-- Meta tags for social media banners, these should be filled in appropriatly as they are your "business card" -->
<!-- Replace the content tag with appropriate information -->
<meta name="description" content="DESCRIPTION META TAG">
<meta property="og:title" content="SOCIAL MEDIA TITLE TAG"/>
<meta property="og:description" content="SOCIAL MEDIA DESCRIPTION TAG TAG"/>
<meta property="og:url" content="URL OF THE WEBSITE"/>
<!-- Path to banner image, should be in the path listed below. Optimal dimenssions are 1200X630-->
<meta property="og:image" content="static/image/your_banner_image.png" />
<meta property="og:image:width" content="1200"/>
<meta property="og:image:height" content="630"/>


<meta name="twitter:title" content="TWITTER BANNER TITLE META TAG">
<meta name="twitter:description" content="TWITTER BANNER DESCRIPTION META TAG">
<!-- Path to banner image, should be in the path listed below. Optimal dimenssions are 1200X600-->
<meta name="twitter:image" content="static/images/your_twitter_banner_image.png">
<meta name="twitter:card" content="summary_large_image">
<!-- Keywords for your paper to be indexed by-->
<meta name="keywords" content="KEYWORDS SHOULD BE PLACED HERE">
<meta name="viewport" content="width=device-width, initial-scale=1">


<title>MuChoMusic: Evaluating Music Understanding in Multimodal Audio-Language Models</title>
<link rel="icon" type="image/x-icon" href="static/images/favicon.ico">
<link href="https://fonts.googleapis.com/css?family=Google+Sans|Noto+Sans|Castoro"
rel="stylesheet">

<link rel="stylesheet" href="static/css/bulma.min.css">
<link rel="stylesheet" href="static/css/bulma-carousel.min.css">
<link rel="stylesheet" href="static/css/bulma-slider.min.css">
<link rel="stylesheet" href="static/css/fontawesome.all.min.css">
<link rel="stylesheet"
href="https://cdn.jsdelivr.net/gh/jpswalsh/academicons@1/css/academicons.min.css">
<link rel="stylesheet" href="static/css/index.css">

<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.5.1/jquery.min.js"></script>
<script src="https://documentcloud.adobe.com/view-sdk/main.js"></script>
<script defer src="static/js/fontawesome.all.min.js"></script>
<script src="static/js/bulma-carousel.min.js"></script>
<script src="static/js/bulma-slider.min.js"></script>
<script src="static/js/index.js"></script>
</head>
<body>


<section class="hero">
<div class="hero-body">
<div class="container is-max-desktop">
<div class="columns is-centered">
<div class="column has-text-centered">
<h1 class="title is-1 publication-title">MuChoMusic: <br/> Evaluating Music Understanding in Multimodal Audio-Language Models</h1>
<div class="is-size-5 publication-authors">
<!-- Paper authors -->
<span class="author-block">
<a href="https://www.upf.edu/web/mtg/about/team-members/-/asset_publisher/l2XuyhfmWvQ5/content/weck-benno/maximized" target="_blank">Benno Weck</a><sup>*</sup><sup>1</sup>,</span>
<span class="author-block">
<a href="https://ilariamanco.com" target="_blank">Ilaria Manco</a><sup>*</sup><sup>2,3</sup>,</span>
<span class="author-block">
<a href="https://www.eecs.qmul.ac.uk/~emmanouilb" target="_blank">Emmanouil Benetos</a><sup>2</sup>,</span>
<span class="author-block">
<a href="https://elioquinton.github.io" target="_blank">Elio Quinton</a><sup>3</sup>,</span>
<span class="author-block">
<a href="https://www.eecs.qmul.ac.uk/~gyorgyf/about.html" target="_blank">George Fazekas</a><sup>2</sup>,</span>
<span class="author-block">
<a href="https://dbogdanov.com" target="_blank">Dmitry Bogdanov</a><sup>1</sup>
</span>
</div>

<div class="is-size-5 publication-authors">
<span class="author-block"><sup>1</sup>UPF, <sup>2</sup>QMUL, <sup>3</sup>UMG<br>ISMIR 2024</span>
<span class="eql-cntrb"><small><br><sup>*</sup>Equal Contribution</small></span>
</div>

<div class="column has-text-centered">
<div class="publication-links">
<!-- Arxiv PDF link -->
<span class="link-block">
<a href="https://arxiv.org/pdf/<ARXIV PAPER ID>.pdf" target="_blank"
class="external-link button is-normal is-rounded is-dark">
<span class="icon">
<i class="fas fa-file-pdf"></i>
</span>
<span>Paper</span>
</a>
</span>

<!-- Data link -->
<span class="link-block">
<a href="https://doi.org/10.5281/zenodo.12709974" target="_blank"
class="external-link button is-normal is-rounded is-dark">
<span class="icon">
<i class="fas fa-database"></i>
</span>
<span>Data</span>
</a>
</span>

<!-- Github link -->
<span class="link-block">
<a href="https://github.com/mulab-mir/muchomusic" target="_blank"
class="external-link button is-normal is-rounded is-dark">
<span class="icon">
<i class="fab fa-github"></i>
</span>
<span>Code</span>
</a>
</span>

<!-- Supplementary PDF link -->
<span class="link-block">
<a href="static/pdfs/supplementary_material.pdf" target="_blank"
class="external-link button is-normal is-rounded is-dark">
<span class="icon">
<i class="fas fa-file-pdf"></i>
</span>
<span>Supplementary</span>
</a>
</span>


<!-- ArXiv abstract Link
<span class="link-block">
<a href="https://arxiv.org/abs/<ARXIV PAPER ID>" target="_blank"
class="external-link button is-normal is-rounded is-dark">
<span class="icon">
<i class="ai ai-arxiv"></i>
</span>
<span>arXiv</span> -->
</a>
</span>
</div>
</div>
</div>
</div>
</div>
</div>
</section>


<!-- Teaser image-->
<section class="hero teaser">
<div class="container is-max-desktop">
<div class="hero-body">
<div class="columns is-centered">
<!-- center the image -->
<img src="./static/images/muchomusic.png" alt="Teaser" class="teaser-image center" width="50%" />
</div>
</div>
</div>
</section>
<!-- End teaser video -->

<!-- Paper abstract -->
<section class="section hero is-light">
<div class="container is-max-desktop">
<div class="columns is-centered has-text-centered">
<div class="column is-four-fifths">
<h2 class="title is-3">Overview</h2>
<div class="content has-text-justified">
<p>
<span class="dnerf">MuChoMusic</span> is a benchmark for evaluating music understanding in multimodal audio-language models. It comprises 1,187 multiple-choice questions, all validated by human annotators, associated with 644 music tracks sourced from two publicly available music datasets, and covering a wide variety of genres. Questions in the benchmark are crafted to assess knowledge and reasoning abilities across several dimensions that cover fundamental musical concepts and their relation to cultural and functional contexts. Each question comes with three distractors composed to test different aspects of language and audio understanding. In the knowledge category, questions probe a model's ability to recognise pre-acquired knowledge across various musical aspects. Questions that test reasoning are instead designed to require the synthesis and analytical processing of multiple musical concepts.
</p>
</div>
</div>
</div>
</div>
</section>
<!-- End paper abstract -->

<section class="section">
<div class="container is-max-desktop">
<div class="columns is-centered has-text-centered">
<div class="column is-four-fifths">
<h2 class="title is-3">Results</h2>
<div class="content has-text-justified">
<p>
Using <span class="dnerf">MuChoMusic</span>, we evaluate five open-source models, three specialised in the music domain and two general-purpose, and find that Qwen-Audio achieves the highest scores on most dimensions.
</p>
<!-- side by side images-->
<div class="columns is-centered">
<img src="./static/images/finegrained_results.png" alt="Finegrained Results" class="teaser-image"
width="60%" height="100%" class="center" />
</div>
<p>
We observe that even the best models can only answer less than 50% of the questions correctly.
Surpringly, out of those considered, models trained on music-specific tasks tend to overall perform worse than those trained on a wider variety of general-audio tasks including speech and everyday sounds.
</p>
<div class="columns is-centered">
<img src="./static/images/results.png" alt="Overview of Results" class="teaser-image"
width="60%" height="100%" class="center" />
</div>
</div>
</div>
</div>
</div>
</section>

<section class="section">
<div class="container is-max-desktop">
<div class="columns is-centered has-text-centered">
<div class="column is-four-fifths">
<h2 class="title is-3">Insights</h2>
<div class="content has-text-justified">
<p>
In an attempt to understand why models perform poorly, we analyse how results change when using only a single distractor (a) or when passing perturbed audio (b).
</p>
<!-- side by side images-->
<div class="columns is-centered">
<img src="./static/images/distractors.png" alt="Experiments with distractors and audio perturbations" class="teaser-image"
width="100%" height="100%" class="center" />
</div>

<p>
From both these experiments, we discover an over-reliance on the language modality pointing to a need for better multimodal integration.
</p>
</div>
</div>
</div>
</div>
</section>



<!--BibTex citation -->
<section class="section" id="BibTeX">
<div class="container is-max-desktop content">
<h2 class="title">BibTeX</h2>
<pre><code>
@inproceedings{weck2024muchomusic,
title={MuChoMusic: Evaluating Music Understanding in Multimodal Audio-Language Models},
author={Weck, Benno and Manco, Ilaria and Benetos, Emmanouil and Quinton, Elio and Fazekas, György and Bogdanov, Dmitry},
booktitle = {Proceedings of the 25th International Society for Music Information Retrieval Conference (ISMIR)},
year={2024}
}
</code></pre>
</div>
</section>
<!--End BibTex citation -->


<footer class="footer">
<div class="container">
<div class="columns is-centered">
<div class="column is-8">
<div class="content">

<p>
This website is licensed under a <a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/" target="_blank">Creative
Commons Attribution-ShareAlike 4.0 International License</a>.
<br>
<br>
It was built using the <a href="https://github.com/eliahuhorwitz/Academic-project-page-template" target="_blank">Academic Project Page Template</a> which was adopted from the <a href="https://nerfies.github.io" target="_blank">Nerfies</a> project page.
</p>

</div>
</div>
</div>
</div>
</footer>

<!-- Statcounter tracking code -->

<!-- You can add a tracker to track page visits by creating an account at statcounter.com -->

<!-- End of Statcounter Code -->

</body>
</html>
1 change: 1 addition & 0 deletions static/css/bulma-carousel.min.css

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading

0 comments on commit cbdf41a

Please sign in to comment.